Distinct social-linguistic processing between humans and large audio-language models: Evidence from model-brain alignment
Year: 2025
Type: conference-paper
Abstract: Voice-based AI development faces unique challenges in processing both linguistic and paralinguistic information.This study compares how large audio-language models (LALMs) and humans integrate speaker characteristics during speech comprehension, asking whether LALMs process speaker-contextualized language in ways that parallel human cognitive mechanisms.We compared two LALMs' (Qwen2-Audio and Ultravox 0.5) processing patterns with human EEG responses.Using surprisal and entropy metrics from the models, we analyzed their sensitivity to speakercontent incongruency across social stereotype violations (e.g., a man claiming to regularly get manicures) and biological knowledge violations (e.g., a man claiming to be pregnant).Results revealed that Qwen2-Audio exhibited increased surprisal for speaker-incongruent content and its surprisal values significantly predicted human N400 responses, while Ultravox 0.5 showed limited sensitivity to speaker characteristics.Importantly, neither model replicated the human-like processing distinction between social violations (eliciting N400 effects) and biological violations (eliciting P600 effects).These findings reveal both the potential and limitations of current LALMs in processing speaker-contextualized language, and suggest differences in social-linguistic processing mechanisms between humans and LALMs.
Authors: Hanlin Wu, Xufeng Duan, Zhenguang Cai
Institution: Chinese University of Hong Kong
Language: English
Subfield: Cognitive Neuroscience
Field: Neuroscience
Domain: Life Sciences
Open Access status: hybrid
OpenAlex ID: W4411113546