Year: 2025
Type: conference-paper
Abstract: Multimodal pre-trained models have demonstrated remarkable capabilities in processing diverse data types, including text and audio. However, their potential for Speech Emotion Recognition (SER) remains relatively underexplored. In this work, we investigate the effectiveness of such models for SER tasks and evaluate the state-of-the-art IISAN framework for efficient fine-tuning. We further improved IISAN by incorporating a dynamic gating mechanism, referred to as IISAN-MOE, to enhance adaptability and performance. Experimental results confirm that multimodal approaches consistently outperform their single-modal counterparts, with IISAN significantly boosting both effectiveness and efficiency. Moreover, the newly proposed IISAN-MOE refines IISAN's original static fusion mechanism, offering a more flexible and advanced solution for multimodal speech emotion recognition.
Authors: Zhiyu Liu, Junchen Fu, Kaiwen Zheng, Joemon M. Jose
Institution: University of Glasgow
Language: English
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: gold
OpenAlex ID: W4410636336