Work

A consistency-constrained fusion network for audio-visual emotion recognition

Year: 2026
Type: article
Abstract: Audio-Visual Emotion Recognition (AVER) from speech and facial expression is a fundamental task in affective computing, yet effectively fusing multimodal features while accounting for both modality consistency and discrepancy remains a challenging problem. Existing approaches either neglect modality discrepancy or model consistency and discrepancy in separate feature spaces. This paper proposes a Consistency-Constrained Fusion Network (CCFNet) that jointly captures unimodal consistency and discrepancy within a unified architecture. To extract modality consistency, two complementary feature fusion models are designed: an attention-based model that performs unimodal self-fusion followed by multimodal cross-fusion, and a Feature-wise Linear Modulation (FiLM) based model that uses facial expression features to modulate speech channel features. To retain modality discrepancy without performance degradation, a consistency encoder constrained by a Central Moment Discrepancy (CMD) loss maps unimodal features into a shared vector space, allowing discrepant features to be preserved rather than discarded as noise. A gate-structure based model aggregates segment-level fused features into video-level representations, and a two-stage training strategy addresses the different learning dynamics between modalities. Experiments on the RAVDESS and CMU-MOSEI datasets demonstrate the effectiveness of the proposed method, achieving recognition accuracies of 79.03% and 68.27% (weighted accuracy), respectively. Ablation studies confirm that each component of the framework contributes to the overall performance.
Language: English
Open Access status: gold
Estimated APC paid: $1,990
OpenAlex ID: W7236665409
FWCI:
Cites: 61
Cited by:
Related to: 10