Answer Length, Not Truth: Re-evaluating White-Box Hallucination Detectors Under Fair Protocols
Year: 2026
Type: preprint
Abstract: White-box hallucination detectors read a language model's internal states to flag hallucinated answers, and recent work reports AUROC above 98%. We re-evaluate two such detectors, MultiHaluDet and ICR Probe, on Mistral-7B and Llama-3.1-8B under one controlled protocol: shared folds, paired bootstrap confidence intervals, and pass/fail criteria fixed before each experiment. On HaluEval QA, answer length alone reaches 0.97 AUROC; when only answers of equal length are compared, MultiHaluDet falls from 0.985 to 0.805-0.815, and its six-classifier stacking stage adds nothing over averaging its deep models. When the models answer TriviaQA and Natural Questions themselves, MultiHaluDet scores below trained token log-probabilities in seven of eight comparisons and ties in the eighth, and ICR Probe scores below them in all eight. A logistic-regression probe on the full hidden state of the last answer token scores higher than MultiHaluDet in all twelve model-setting-label comparisons. Combined with token log-probabilities, it beats MultiHaluDet combined with the same log-probabilities in all eight realistic comparisons, with every confidence interval above zero. MultiHaluDet's training-seed variance (s.d. <=0.005) is far smaller than these gaps. We release our code and evaluation protocol.
Author: Khan, Mohammed Abdul Jawad Ali
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7224512537