Discovering and Causally Validating Emotion-Sensitive Neurons in Large Audio-Language Models
Year: 2026
Type: conference-paper
Abstract: Emotion is a central dimension of spoken communication, yet, we still lack a mechanistic account of how modern large audio-language models (LALMs) encode it internally.We present the first neuron-level interpretability study of emotion-sensitive neurons (ESNs) in LALMs and provide causal evidence supporting the existence of such units in Qwen2.5-Omni,Kimi-Audio, and Audio Flamingo 3. Across these three widely used open-source models, we compare frequency-, entropy-, mean-deviation-, and contrast-based neuron selectors on multiple emotion recognition benchmarks.Using inference-time interventions, we reveal a consistent emotion-specific signature: deactivating neurons selected for a given emotion disproportionately degrades recognition of that emotion while largely preserving other classes, whereas targeted steering amplifies these units to bias predictions toward the target emotion.These effects arise with modest amounts of identification data and scale systematically with intervention strength.We further observe that ESNs exhibit non-uniform layerwise clustering with partial cross-dataset transfer.Taken together, our results offer a causal, neuron-level account of emotion decisions in LALMs and highlight targeted neuron interventions as an actionable handle for controllable affective behaviors.
Authors: Xiutian Zhao, Björn Schuller, Berrak Sisman
Institutions: Johns Hopkins University, Imperial College London
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: National Science Foundation
OpenAlex ID: W7166800298