Automatic music emotion tag generation for short-video platforms: a self-supervised audio-lyric fusion network with valence-arousal regularization
Year: 2026
Type: article
Abstract: Background music serves as a primary carrier of emotional narrative on short-video platforms, where hundreds of millions of user-generated clips are uploaded daily and organized, searched, and recommended largely through metadata. Emotion-related tags, however, are still produced mainly by manual annotation or by generic audio taggers that were never designed for fifteen-to-sixty-second clips recorded in uncontrolled environments. This paper addresses the problem of automatic music emotion tag generation for short-video platforms, a setting characterized by truncated clips, ubiquitous audio degradation, and unreliable lyric availability. To tackle these challenges, we propose SALF-Net, a self-supervised audio-lyric fusion network specifically designed to be robust to modality absence and input noise. The audio branch adapts a music-domain self-supervised transformer encoder to clip-level inputs through masked segment sampling and statistics pooling, while the lyric branch encodes automatically transcribed or metadata-provided lyrics with a pretrained language model. A gated cross-modal fusion module with modality dropout keeps the system functional when lyrics are missing, which is common for instrumental tracks and noisy recordings. A multi-task objective jointly optimizes multi-label tag prediction and valence-arousal regression, so that the psychological structure of emotion space acts as a regularizer for tag geometry. Experiments on the mood and theme subset of MTG-Jamendo, on the DEAM dimensional emotion benchmark, and on a newly constructed short-video test collection of 30,000 clips show that the proposed model reaches a precision-recall AUC of 0.389 and an ROC-AUC of 0.921, outperforming strong self-supervised baselines by a clear margin, while a distilled student variant preserves 95% of this quality at 2% of the teacher’s latency. The results indicate that accurate, robust, and deployable emotion tagging is feasible at short-video scale without relying on platform-private annotation budgets.
Source: Scientific Reports
Authors: Zhiqi Liu, Huangfeng Hong
Institutions: Jimei University, Xiamen University
Language: English
Topic: Music and Audio Processing
Subfield: Signal Processing
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7223068168