A lightweight natural language conditioning framework for controllable polyphonic music generation
Year: 2026
Type: article
Abstract: Controllable symbolic music generation remains a challenging problem, as many generative models can compose musically plausible sequences but fail to consistently follow user-defined stylistic prompts over long durations. This paper presents a lightweight natural language conditioning framework for polyphonic music generation that converts free-form text prompts into structured control tokens representing genre, tempo, and note density. Unlike computationally intensive text-to-audio systems, the proposed approach uses an efficient rule-based translation layer that injects deterministic constraints directly into the generation context. To examine the impact of architecture on long-term prompt adherence, two autoregressive models were evaluated: a Long Short-Term Memory (LSTM) network and a decoder-only Transformer. Both models were pre-trained on 11,000 MIDI files and later adapted through transfer learning after vocabulary expansion to incorporate the proposed conditioning tokens. While the cross-entropy converged similarly during training, the two architectures produced structurally different outputs. During the test, LSTM exhibited temporal collapse, overproducing note density (47.3 notes/s) and implausible polyphony (86.4%). In contrast, the Transformer generated more balanced outputs with note density of 7.1 notes/s and polyphony of 28.1%, substantially closer to the human baseline. These results demonstrate that token-level loss alone is insufficient for evaluating controllable music generation quality. More importantly, the results indicate that the Transformer architecture, using self-attention, provides stronger preservation of global prompt semantics and long-range musical coherence than the recurrent LSTM baseline under the evaluated experimental settings.
Source: Discover Artificial Intelligence
Authors: Animesh Mishra, D. Suma
Institution: Manipal Academy of Higher Education
Language: English
Subfield: Music
Field: Arts and Humanities
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $1,590
OpenAlex ID: W7235692176
