Acoustic feature decoupling and pre-trained language model integration for music-to-text cross-modal generation
Year: 2026
Type: article
Abstract: Translating musical content into coherent natural language descriptions is a difficult cross-modal generation problem, mainly because acoustic representations entangle multiple musical attributes and a wide semantic gulf separates audio from text. We present a framework that brings together acoustic feature decoupling and large-scale pre-trained language models for music captioning. At its core, a variational autoencoder-based module factorises audio representations into three semantically distinct subspaces-content, style, and emotion-guided by mutual information minimisation together with orthogonality constraints. A multi-granularity alignment mechanism then bridges the decoupled acoustic features and linguistic representations through both global and local contrastive learning. The pre-trained language model decoder, adapted via prefix tuning, retains its accumulated linguistic knowledge while accepting multimodal conditioning. Across three benchmarks-MusicCaps, Song Describer, and LP-MusicCaps-the framework delivers competitive performance relative to recent baselines, with relative gains of 18.9% in BLEU-4 and 4.4% in BERTScore over CLAP-Cap on MusicCaps. Ablation studies confirm that each component contributes meaningfully to generation quality, while human evaluation validates superior fluency, relevance, informativeness, and diversity of generated descriptions.
Source: Scientific Reports
Author: Jiaying Yu
Institution: Hanyang University
Language: English
Topic: Music and Audio Processing
Subfield: Signal Processing
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7166697882
FWCI:
Cites:
Cited by:
Related to: