Work

Multimodal emotion recognition and feedback adaptation for personalized music therapy

Year: 2026
Type: article
Abstract: Effective music therapy depends on dynamic matching among an individual’s emotional state, music emotion attributes, and intervention feedback. Existing intelligent music therapy studies have focused mainly on system implementation or recommendation performance, while paying insufficient attention to how emotion recognition informs therapeutic decision-making, how music features map onto regulation goals, and how user feedback revises subsequent interventions. Based on the valence-arousal emotion model, this study develops a personalized music therapy framework driven by multimodal emotion recognition. The framework maps the user’s current emotional state, target therapeutic state, and music emotion features into a unified space, and models the relationships among current-state congruence, target-state approach, user preference, and historical feedback through a therapeutic utility function. In implementation, a cross-modal Transformer is employed to fuse facial, speech, and physiological signals for user emotion representation; VGGish with dual-task learning is used to extract music valence and arousal features; and a hybrid strategy integrating content matching, collaborative filtering, and DQN-based feedback optimization is used to generate personalized music intervention plans. Experimental results show that the proposed method achieved 90.1% emotion-recognition accuracy and an F1 score of 89.6%. For music emotion prediction, the valence and arousal MAEs were 0.096 and 0.101, respectively, with a correlation coefficient of 0.861. The recommendation module achieved HR@10 and NDCG@10 values of 84.2% and 80.5%, respectively. In the short-term affective evaluation, the mean reduction in distance from the current state to the predefined target state was 0.214 for the proposed estimator, compared with 0.186 for Tensor Fusion. These results support the computational feasibility and adaptability of the framework, while not constituting evidence of clinical therapeutic efficacy.
Author: Kejing Chen
Language: English
Open Access status: gold
Estimated APC paid: $2,690
OpenAlex ID: W7237360987
FWCI:
Cites:
Cited by:
Related to: