Predicting reference matches from generator and reader states
Year: 2026
Type: preprint
Abstract: Predicting reference matches from generator and reader states: Stage P exploratory pilot report (Version 1.0) A published pilot study reported that a language model's hidden state predicts its own eventual answer better than a second model reading the same text. This report tests a related question on factual recall: can a model's own state predict which of its sampled answers a reference matcher will reject, better than an outside reader can? Qwen2.5-1.5B-Instruct generated 16 justification-then-answer responses to each of 120 factual questions in the T-08R powered ladder. The analysis uses the 999 committed generations on the 64 questions that produced both matching and non-matching answers. A linear probe read the generator's state (Self) and a second probe read Qwen2.5-3B-Instruct on the identical text (Reader), at a depth fixed before fitting: block 14 of 28 and block 18 of 36. Probes were scored on held-out questions by within-question pooled AUC. The smallest meaningful Self advantage, +0.03 AUC, was fixed before any probe was fitted. At the fixed depth, the paired 95% intervals rule out a +0.03 Self advantage at the eighth justification token (Self minus Reader −0.008, interval −0.036 to +0.019) and at the end of the justification (−0.035, interval −0.067 to −0.002). Immediately before the formal answer the difference was +0.013 (interval −0.025 to +0.051), so an advantage of that size remains open there. The same three conclusions hold after removing the 24 generations on which two string matchers disagreed. Adding the Self score to the generator's own output probabilities did not measurably improve prediction. A descriptive layer sweep shows that the fixed depth sits just before predictive signal appears in both models. In later layers the larger Reader scores at least as high as Self. That sweep was not pre-specified and has no intervals. Scope and limits. This is an exploratory pilot on one model pair and 64 questions. The endpoint is a string matcher's verdict, and independent validation of factual correctness remains open. Historical token sequences were reconstructed from saved text and screened by token count. A confirmatory study with fresh questions and a comparable-size reader is planned and has produced no data. Files. The report (PDF) and Stage_P_Reproducibility.zip, which contains the held-out predictions, per-question win counts, fold assignments, run configuration, the analysis notebook, the dated pre-fit amendment and layer curves. Running python reproduce_tables.py rebuilds both report tables on a CPU without model inference. A full probe refit additionally requires the 2.8 GB state capture, which is not included. Related records. T-08R ladder data: 10.5281/zenodo.21707044. Ladder report: 10.5281/zenodo.21714854.
Author: Joshua Duran
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7237354575
Cites:
Cited by:
Related to:
