THAU-UPM at EmoSPeech-IberLEF2024: Efficient Adaptation of Mono-modal and Multi-modal Large Language Models for Automatic Speech Emotion Recognition
Year: 2024
Type: conference-paper
Abstract: Automatic Speech Emotion Recognition (SER) is a pivotal task across domains such as psychology, healthcare, or human-computer interaction. This study, conducted within the framework of the EmoSpeech challenge at IberLEF 2024, explores efficient adaptation techniques for mono-modal and multi-modal large language models (LLMs) for SER in Spanish. The challenge includes two tasks: one that relies solely on textual transcriptions and another that integrates audio signals. For the text-only task, we fine-tune the multilingual Gemma-2B model using the Low-Rank Adaptation (LoRA) method, optimizing low-rank adaptation parameters. For the multi-modal task, we employ Qwen-Audio-Chat, enhancing its performance using the LoRA technique, and we propose a novel Whisper-Gemma model that integrates Whisper-large-v3 audio encoder with the Gemma LLM, training only a projection layer. The metrics obtained as a result of the experimentation process demonstrate the potential of both approaches. The fine-tuned Gemma-2B model achieves an f1-macro score of 0.6094 on the text-only task, while the Qwen-Audio-Chat model reaches 0.8248, indicating significant improvements in emotional recognition capabilities when combining both modalities. Additionally, the Whisper-Gemma model achieves a competitive f1-macro score of 0.7904, underscoring the effectiveness of using pre-trained audio encoders and smaller LLMs in SER tasks. These findings highlight the value of parameter-efficient fine-tuning methods and the integration of robust audio encoders with LLMs to enhance SER performance.
Source: Hispana
Language: English
Topic: Emotion and Mood Recognition
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G1175115578, G1435188718, G1855990336, G210041235, G2114305129, G2262748287, G2571741393, G2601397997, G279363373, G300979063, G3045744994, G34346518, G3480869486, G3681454997, G4042783231, G425668148, G430027085, G430777504, G5080475149, G5421974903, G6234352027, G6238863487, G7434936867, G750425203, G7702339200, G99296377
OpenAlex ID: W7161618651
