Knowing When Retrieval Has Failed: A Cheap, Unsupervised Signal That Generalizes Across Domains
Year: 2026
Type: preprint
Abstract: Retrieval-augmented generation fails silently: when the retriever returns irrelevant passages, the language model answers anyway, and in high-stakes domains such as mental health question answering a fluent answer built on bad context can be harmful. For anyone running RAG who needs to catch these failures before generation, we ask whether they can be detected without supervision, without a second model, and at negligible cost, using only the dense retrieval score distribution. The central contribution is transfer. Two pre-registered signals, a shape contrast on the ranked list and a calibration of the top score against a per-query random null, are frozen and evaluated unchanged on three natural-query BEIR benchmarks, including a finance corpus outside the biomedical source domain. Both satisfy a pre-registered generalization criterion on all three, with oriented AUROC of 0.70 to 0.81 and a family-wise corrected p near zero. Precision-recall calibrates the expectation at 1.5 to 2.8 times the failure prevalence, which is a useful and transferable abstention gate and not a near-perfect classifier. Under a lexical-overlap control, and against a panel of nineteen signals given identical treatment, the calibration signal is the most robust and the most consistent, which makes it the best deployment choice. The highest raw numbers, up to 0.91, come from the realistic discovery corpus, but signals such as score entropy do not survive the lexical control, so we lead with the ones that transfer. Beyond the detector, we audit the failure target itself and put the signal to work. An annotation study shows that the gold-based target is a faithful signal of gold-recall failure but a pessimistic proxy for utility, because a useful non-gold passage is almost always present even when the gold is missing. Thresholding the signal to abstain yields favorable risk-coverage behavior. With a domain reader the abstention is non-inferior to answering every query and slightly improves accuracy, and the verdict of the signal tracks, per query, how much the context improves the answer. Two domain refinements do not pay off, and both are reported: lay harm annotation does not reach reliability, and scaling abstention by severity does not beat the plain signal. The result is a cheap, transferable foundation for abstention in high-stakes retrieval pipelines, with a stated account of which components earn their place and which do not. Pre-print in progress. The file is restricted while the manuscript is under advisor review. Access is granted by share link.
Author: Gustavo Dias, Apolinário
Institution: Universidade Federal de Minas Gerais
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7210273558
Cites:
Cited by:
Related to: