A mamba-driven transformer method for multimodal emotion recognition optimization
Year: 2026
Type: article
Abstract: Emotion recognition, a core technology of affective computing, directly impacts human–computer interaction. However, existing CNN and Transformer-based models have limitations: CNNs struggle with long sequence data due to local receptive fields, while Transformers face quadratic computational complexity with increasing sequence lengths, limiting complex emotion state recognition. To address these challenges, we propose the TriModalMam model, a multimodal emotion prediction framework. TriModalMam integrates the advantages of the Mamba architecture, optimizing emotion feature extraction and fusion by combining text, audio, and visual features. The model uses the Monomodal Sequential Mamba (MSM) module for deep feature extraction, projects modality features into a similarity subspace through shared encoders, and optimizes feature representations with Mamba-driven Multi-modal Extraction (MMFE). Cross-Modal Interaction (CMI) enhances information flow and interaction between modalities, and the fused features are processed by a Transformer encoder and MLP network for emotion prediction. Experiments on the CMU-MOSI and CMU-MOSEI datasets show that TriModalMam outperforms traditional CNN and Transformer models. On CMU-MOSI, the seven-class accuracy reaches 87.34, with a correlation of 0.855; on CMU-MOSEI, accuracy reaches 87.45, with correlation of 0.891. The model has only 110M parameters, balancing high performance and low computational complexity.
Source: Alexandria Engineering Journal
Authors: Dongwei Ren, Song Guo
Institution: Ordos Institute of Technology
Language: English
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $1,900
OpenAlex ID: W7162127937