Work

A lightweight natural language conditioning framework for controllable polyphonic music generation

Year: 2026
Type: article
Abstract: Controllable symbolic music generation remains a challenging problem, as many generative models can compose musically plausible sequences but fail to consistently follow user-defined stylistic prompts over long durations. This paper presents a lightweight natural language conditioning framework for polyphonic music generation that converts free-form text prompts into structured control tokens representing genre, tempo, and note density. Unlike computationally intensive text-to-audio systems, the proposed approach uses an efficient rule-based translation layer that injects deterministic constraints directly into the generation context. To examine the impact of architecture on long-term prompt adherence, two autoregressive models were evaluated: a Long Short-Term Memory (LSTM) network and a decoder-only Transformer. Both models were pre-trained on 11,000 MIDI files and later adapted through transfer learning after vocabulary expansion to incorporate the proposed conditioning tokens. While the cross-entropy converged similarly during training, the two architectures produced structurally different outputs. During the test, LSTM exhibited temporal collapse, overproducing note density (47.3 notes/s) and implausible polyphony (86.4%). In contrast, the Transformer generated more balanced outputs with note density of 7.1 notes/s and polyphony of 28.1%, substantially closer to the human baseline. These results demonstrate that token-level loss alone is insufficient for evaluating controllable music generation quality. More importantly, the results indicate that the Transformer architecture, using self-attention, provides stronger preservation of global prompt semantics and long-range musical coherence than the recurrent LSTM baseline under the evaluated experimental settings.
Language: English
Subfield: Music
Open Access status: gold
Estimated APC paid: $1,590
OpenAlex ID: W7235692176
FWCI:
Cites: 16
Cited by:
Related to: 10