On the Nuances of Multimodal Laughter Detection: Where, Why, and When Visual Insight Augments Audio in Classification
Year: 2025
Type: article
Abstract: This paper explores the impact of visual input on traditional audio classification, focusing on the analysis of audiovisual data compilation and the interaction between modalities in laughter detection. While assessing user reactions to measure potential engagement is distinct from evaluating a content’s entertainment value, incorporating user-centered audio and visual cues enhances the contextual understanding of stimuli and the completeness of the emotional response. Among the results, making use of facial expressions augments the audio-only prediction accuracy with up to 26% when cross-attention mechanisms are employed. To evaluate the robustness of our proposed approach, experiments were systematically conducted under unimodal conditions, omitting each modality individually to asses its impact on performance. As audio and visual insights yield unbalanced contributions to the user’s overall emotional profiling, the accuracy gain of visual features on top of the audio setup provides evidence on the classifier’s ability to capture the nuances of human laughter behaviors.
Source: IEEE Access
Authors: Liviu-Alexandru Marina, Corina Dimitriu
Institutions: Xperi (United States), Alexandru Ioan Cuza University
Language: English
Topic: Emotion and Mood Recognition
Subfield: Experimental and Cognitive Psychology
Field: Psychology
Domain: Social Sciences
Open Access status: gold
Estimated APC paid: $2,075
OpenAlex ID: W4411358757