End-to-End Multimodal Emotion Recognition with Deep Temporal and Cross-Modal Feature Integration
Year: 2026
Type: article
Abstract: : Emotion recognition has advanced significantly with the adoption of deep learning methods, yet reliable inference of affective states remains challenging under real-world conditions characterized by noise, occlusion, and expressive ambiguity. These limitations are particularly evident in unimodal systems that rely on a single source of affective information. To address this challenge, this study proposes a novel end-to-end multimodal framework for temporal emotion recognition that jointly models facial and vocal cues within a unified deep learning architecture. The proposed framework integrates deep residual networks for spatial and spectral feature encoding with bidirectional long short-term memory networks for sequence-level temporal modeling. Audio signals are represented using Mel-Frequency Cepstral Coefficients, while facial information is extracted from video frames, with both modalities processed using a shared ResNet-50 backbone to ensure consistent high-level representations. To enhance multimodal interaction, the framework incorporates attention mechanisms that refine modality-specific temporal features and explicitly align audio and visual representations prior to classification. The model is evaluated on the RAVDESS and CREMA-D benchmark datasets using strict subject-disjoint cross-validation to ensure unbiased assessment of generalization performance. Experimental results demonstrate that the proposed approach achieves classification accuracies of 91.22 percent on RAVDESS and 87.32 percent on CREMA-D, outperforming recent multimodal methods evaluated under comparable conditions. Confusion-matrix-based analyses further indicate improved discrimination among emotionally overlapping categories. These results demonstrate that jointly modeling deep spatial representations, temporal dynamics, and adaptive cross-modal interaction yields robust emotion recognition under realistic variability. The proposed framework provides a transparent and extensible foundation for future research in multimodal affective computing and emotion-aware intelligent systems.
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: hybrid
OpenAlex ID: W7204774007
FWCI:
Cites:
Cited by:
Related to: