Right Document, Wrong Sentence: A Pre-registered Abstention Study of a Local Multi-Document Research Assistant, in Which a Frozen Threshold Refused 90 Percent of Unanswerable Questions and the Extracted Sentence Held the Answer Only 41 Percent of the Time
Year: 2026
Type: preprint
Abstract: A retrieval tool that answers questions from a local document folder faces two distinct failure modes: it can answer questions the documents cannot support, and it can point at the right document while returning the wrong sentence. This study measures both, under rules frozen in advance. The tool under test is a local research assistant built by the author: it chunks a folder of documents into 120-word windows, embeds them locally with minishlab/potion-base-8M through model2vec, retrieves the single top chunk by cosine similarity, returns the sentence inside that chunk most similar to the question with a citation, and abstains when the top chunk scores below a frozen threshold of T = 0.40. The corpus (16 self-written technical documents, 5,124 words, 62 chunks), the question battery (30 in-corpus questions with gold documents and gold phrases, 20 out-of-corpus questions including deliberate vocabulary traps), the threshold, and three hypotheses with numeric verdict lines were committed together in a pre-registration (2026-11-15) before the tool's measurement code was written. All three hypotheses were CONFIRMED. H1: 29 of 30 in-corpus questions (0.967) were answered with the gold document cited, against a frozen line of 0.80. H2: 18 of 20 out-of-corpus questions (0.900) were refused, against a line of 0.80 and against the reference failure that motivated the study, an earlier system in the same programme that abstained on only 0.167 of its adversarial questions. H3: median per-question latency was 0.0013 s over 250 measurements, against a line of 2.0 s. The frozen secondary measure exposes the weak stage: of the 29 answered in-corpus questions, only 12 (0.414) returned the sentence containing the gold phrase, so the citation is the reliable product and the extracted sentence is a pointer. The failures are three and all sit in the overlap band between the two score distributions (in-corpus top scores 0.397 to 0.762, out-of-corpus 0.090 to 0.507): one in-corpus abstention at 0.3967, and two vocabulary traps answered at 0.4162 and 0.5071. One correction to the study record is reported in full: the repository's RESULTS.md states that the abstained question's correct chunk was the top retrieval result, but re-execution with the threshold disabled shows the top chunk came from a different document, so the abstention also masked a wrong-document retrieval. The claims are deliberately narrow: one small self-written corpus, one embedding model, one machine, one threshold. Within those bounds, a threshold frozen before the fact produced a tool that refuses most of what it cannot support, where the reference system in the motivating record refused almost nothing, at a cost of one in-corpus refusal, and the remaining defect is sentence selection, not retrieval or refusal.
Author: Ushno, Jesun Ahmad
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7235962271
