Work

Advancing Audio Emotion and Intent Recognition with Large Pre-Trained Models and Bayesian Inference

Year: 2023

Type: conference-paper

Abstract: Large pre-trained models are essential in paralinguistic systems, demonstrating effectiveness in tasks like emotion recognition and stuttering detection. In this paper, we employ large pre-trained models for the ACM Multimedia Computational Paralinguistics Challenge, addressing the Requests and Emotion Share tasks. We explore audio-only and hybrid solutions leveraging audio and text modalities. Our empirical results consistently show the superiority of the hybrid approaches over the audio-only models. Moreover, we introduce a Bayesian layer as an alternative to the standard linear output layer. The multimodal fusion approach achieves an 85.4% UAR on HC-Requests and 60.2% on HC-Complaints. The ensemble model for the Emotion Share task yields the best ρ value of .614. The Bayesian wav2vec2 approach, explored in this study, allows us to easily build ensembles, at the cost of fine-tuning only one model. Moreover, we can have usable confidence values instead of the usual overconfident posterior probabilities.

Source: ACM International Conference on Multimedia (ACM MM)

DOI: 10.1145/3581783.3612848

Authors: Dejan Porjazovski, Yaroslav Getman, Tamás Grósz, Mikko Kurimo

Institution: Aalto University

Language: English

Topic: Emotion and Mood Recognition

Subfield: Experimental and Cognitive Psychology

Field: Psychology

Domain: Social Sciences

Open Access status: hybrid

Funders: Academy of Finland, Aalto-Yliopisto

Awards: Common Language Resources and Technology Infrastructure, G8125348452, Understanding speech and scene with ears and eyes