Year: 2026
Type: conference-paper
Abstract: Hallucinations in Speech Large Language Models (SpeechLLMs) pose significant risks, yet existing detection methods typically rely on goldstandard outputs that are costly or impractical to obtain.Moreover, hallucination detection methods developed for text-based LLMs do not directly capture audio-specific signals.We investigate four attention-derived metrics: AU-DIORATIO, AUDIOCONSISTENCY, AUDIOEN-TROPY, and TEXTENTROPY, designed to capture pathological attention patterns associated with hallucination, and train lightweight logistic regression classifiers on these features for efficient inference-time detection.Across automatic speech recognition and speech-to-text translation tasks, evaluations on Qwen-2-Audio and Voxtral-3B show that our approach outperforms uncertainty-based and prior attention-based baselines on in-domain data, achieving improvements of up to +0.23 PR-AUC, and generalises to out-of-domain ASR settings.We further find that strong performance can be achieved with approximately 100 attention heads, improving out-ofdomain generalisation compared to using all heads.While effectiveness is model-dependent and task-specific training is required, our results demonstrate that attention patterns provide a valuable tool for hallucination detection in SpeechLLMs.
Institution: University of Edinburgh
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166813386