Shuffling Augmented Decoupled Features for Multimodal Emotion Recognition
Year: 2025
Type: article
Abstract: Multimodal emotion recognition (MER) aims to identify human emotions using data from multiple modalities. Despite promising advances in previous MER methods, their performance remains limited due to the small size of available datasets, a result of the challenges in collecting multimodal data. While data augmentation can address this issue, generating augmented multimodal data without altering the underlying emotional meaning remains particularly challenging. To tackle this problem, we introduce a decoupled feature augmentation method that automatically learns multimodal feature variations in a decoupled feature space for MER. Specifically, we decompose multimodal features into modality-invariant and modality-specific components and then augment each component within the decoupled feature space across multiple modalities. Our method ensures harmonious feature integration across modalities by collectively generating multimodal features, while preserving the overall semantics of the data through the decoupling scheme. To enhance model generalization and stability, we propose a learning strategy that gradually incorporates more diverse information by using a combined set of original and augmented decoupled features. Comprehensive experiments on two MER benchmarks demonstrate that our method outperforms or is comparable to several baseline methods.
Source: IEEE Access
Author: Sunyoung Cho
Institution: Sookmyung Women's University
Language: English
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $2,075
OpenAlex ID: W4410639280