Work

Leveraging dynamic feature fusion of self & cross-attention for robust multimodal emotion recognition

Year: 2025

Type: article

Abstract: Recent multimodal emotion recognition (MER) approaches often assume that fusing audio and video modalities through cross-attention inherently yields superior feature representations for MER. However, our analysis reveals that different emotions manifest more prominently in specific modalities, leading to frequent misclassifications when an unsuitable modality dominates. Building on this, we propose a framework that dynamically adjusts each modality’s contribution using weighted integration of self & cross-attention for emotion, called WISE-Net. In addition, as certain emotions benefit from merging complementary audio-video information, we introduce a cross-label emotion matching task to explicitly capture fine-grained emotion-related distinctions across modalities. Experiments confirm that WISE-Net surpasses state-of-the-art methods. Ablation studies highlight the significance of balancing self-attention for modality expressiveness and cross-attention for emotion-aware feature fusion.

Source: ICT Express

DOI: 10.1016/j.icte.2025.08.005

Authors: Jian Kim, Hyunoh Kang, Haneul Pyeon, Dahuin Jung

Institution: Soongsil University

Language: English

Topic: Emotion and Mood Recognition

Subfield: Experimental and Cognitive Psychology

Field: Psychology

Domain: Social Sciences

Open Access status: gold

Estimated APC paid: $1,500

Funders: National Research Foundation of Korea, Ministry of Science and ICT, South Korea, Institute for Information and Communications Technology Promotion