Work

The Offline Core Is Perfect Until the Wording Changes: An Exploratory Study of a Standard-Library TF-IDF Assistant with Cited Answers and Threshold Refusal, and a Modelled Local against Cloud Comparison

Year: 2026
Type: preprint
Abstract: Can a fully offline assistant, built from the Python standard library alone and running no language model at all, answer questions over private documents with cited answers and correct refusals? This study built one and measured it, and it states its status up front: the study is exploratory, not pre-registered. The hypotheses were written on 2026-11-07, after the main runs of 2026-11-05, and the paper reports them as post hoc. The system ingests a synthetic corpus of 24 documents (144 sentences, 1,472 words), splits it into 72 overlapping chunks of 30 words, indexes the chunks with TF-IDF, answers with the best two sentences of the top document copied verbatim, each with its own citation, and refuses any question whose best chunk score falls below a threshold of 0.30. On the 24 labelled questions the scores are perfect: top-1 document accuracy 20 of 20 (Wilson 95 percent interval 0.839 to 1.000), chunk recall at 3, 20 of 20, first answer sentence accuracy 20 of 20, and refusal precision and recall 4 of 4 with no false refusals; a leave-one-out check, in which each question is scored against a threshold fitted on the other 23, also refuses 4 of 4 with no false refusals (fitted thresholds 0.335 to 0.356), though the refusal recall interval is wide (0.510 to 1.000) because only 4 no-evidence questions exist. The misses are the finding. On a 16-question hard set written to avoid the documents' wording, the same system at the same threshold answered only 2 of 10 synonym paraphrases correctly (8 refused, although the right document was the top chunk for 6 of 10) and refused only 2 of 6 cross-topic distractors, so the separation does not survive rewording (H3, not supported). A baseline comparison cannot rank the retrieval settings on this corpus: TF-IDF over chunks, BM25 over the same chunks, and TF-IDF over whole documents all score 20 of 20, and across a 24-setting chunk sweep only refusal moves (H4, not testable here). Latency stays interactive as the corpus grows, from a mean of 0.143 ms at 72 chunks to 13.7 ms at 7,200 chunks (H5, supported to 7,200 chunks). The local against cloud half of the question is answered only in part: no local language model ran (Ollama was unavailable on the study machine) and no cloud API was called, so cloud cost is modelled from published pricing checked on 2026-11-05 (0.000031 USD per query at budget rates to 0.000621 USD at flagship rates, at a stated word-based token estimate), cloud latency is one labelled assumed range, and cloud quality is a capability comparison, not a measurement. What the study establishes is narrow and reproducible: an offline extractive core with citations and a refusal rule works end to end, its recorded results regenerate from the committed code and corpus, and its ceiling is exact token matching, now measured rather than asserted.
Language: English
Open Access status: green
OpenAlex ID: W7236052920
Cites:
Cited by:
Related to: 10