Multilingual Speech Emotion Recognition combining Audio and Text Information using Small Language Models
Year: 2024
Type: conference-paper
Abstract: In this work, we present a multimodal Small Language Model (SLM) architecture designed for multilingual Speech Emotion Recognition (SER). Our approach integrates a transformer- based audio encoder with a SLM, using a linear projection layer that bridges audio inputs with textual comprehension. This in- tegration enables the SLM to effectively process and understand spoken language, enhancing its capability to recognize emo- tional nuances. We experiment with various state-of-the-art (SoTA) SLMs and evaluate them across five different datasets representing a variety of European languages: German, Por- tuguese, Italian, Spanish, and English. By leveraging both au- dio signals and their corresponding transcriptions, our model achieves comparable performance in SER tasks for each lan- guage with respect to SoTA models. Our results demonstrate the robustness of our archit
Source: Hispana
Authors: Bellver Soler, Jaime, Rodríguez Cantelar, Mario, Estecha Garitagoitia, Marcos Santiago, Córdoba Herralde, Ricardo de, D'Haro Enríquez, Luis Fernando
Language: English
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7058415054
FWCI:
Cites:
Cited by:
Related to: