Get started
Work

Data Reuse, Not Selection: A Code-Verified Audit of Optimism in Hidden-State Hallucination-Detection Pipelines

Year: 2026
Type: preprint
Abstract: Status: Accepted as a poster at the NeurIPS 2026 Workshop on Trustworthy AI Evaluation (TAE). This is the camera-ready version. An extended version is under review at TMLR. When a checkpoint, layer, threshold, or hyperparameter is selected on data later reused to report performance, the reported number is optimistic. This is well documented for discrimination metrics such as AUROC. In the settings we test, evaluating on reused data at a selected configuration also costs calibration – how reliable a system's stated confidence is, not just how well it ranks – and degrades the operating point a selective-prediction threshold is believed to buy. Selecting one layer by argmax over 32 candidates, with no fitting on the evaluation data at all, inflates achieved risk by 10–16% relative to the believed budget: at a 70% coverage target over 100,000 queries a day, 1,082 additional unflagged errors daily (BCa 95% [435,1718]), priced at the coverage each arm actually achieves. That is one 700-item sample, and its intervals are split-conditional; on 25 independent samples (17,500 TriviaQA questions) the cost is established at population level — 0.85 extra unflagged errors per 100 answered queries at 50% coverage — while the selected layer is genuinely better held out. We separate calibration from discrimination rather than inferring it: under a Murphy decomposition the reliability term carries the effect (+0.0046, BCa [+0.0027,+0.0065]) while resolution does not. The ECE gap holds in all eight binning settings and a binning-free calibration slope agrees (0.68 reused versus 0.62 held out). We do not attribute the gap to the selection step: a placebo at unselected layers absorbs roughly half of it, and the remaining selection-specific increment does not clear its own mechanical null, which is positive in expectation because the selected index maximises the same criterion the gap contains. On a second harness, two model families, the corrected test says the selected layer transfers on every metric and the cost left is small. Nine of ten cells are established per-cell, six cleanly and three resting on a non-negativity our own method section disqualifies, and all nine survive family-wise correction. Measured severity is a property of a (mechanism, metric, control) triple: one cell spans 1.44× across two admissible controls. We release the taxonomy, three pinned third-party verifications, a checklist extending Kapoor & Narayanan and REFORMS, and a negative result – a regex scanner automating four of its five items finds zero true positives across seven repositories and misses both leaks it was built to catch. Our object of study is the evaluation protocol, not the model. Code, result files and a verifier for every number in the paper: doi:10.5281/zenodo.23624982 (GitHub: selection-leakage-calibration-audit).
DOI: 10.5281/zenodo.23624979
Language: English
Open Access status: green
OpenAlex ID: W7223445378
Cites:
Cited by:
Related to: