Does the Direction Track the Process? A Preregistered Validity Battery for Published Hidden-State Directions
Year: 2026
Type: preprint
Abstract: A linear direction in a language model's hidden states is usually defended by two facts: it separates two sets of prompts that differ by design, and adding it to the hidden state changes the output in the expected way. Neither fact shows that the direction follows the process it is named after. Earlier papers in this series found a published-style direction that separated prompt types in twelve models while tracking neither the computation it was named for nor its supposed output, and that mostly toggled reply format. Here we package the checks that exposed it as a preregistered battery and apply it uniformly to published directions on open models of at most 14B parameters. The battery asks whether a direction (A) follows a per-item target measured from what the model actually does, (B) still separates its classes on stimuli matched for template, length and vocabulary, (C) beats two directions built from the same calibration prompts without their labels, (D) steers beyond random pushes that change the output distribution by the same KL, and (E) hands its effect on to later layers rather than acting as an output bias; an oracle direction built from the target itself tells whether a failure is informative. We apply the battery to nine published directions on eight open models (27 direction–model pairs, plus a second item set for one direction). The two controls pass: the refusal and truth directions are valid drivers overall, although refusal acts as an output bias in OLMo-2. The compute–retrieve axis reproduces the earlier causal null. None of the six process directions receives a valid verdict. Four of them (LiReFs, backtracking, the sycophancy persona vector, true thinking) are target not linear: an oracle built from the process itself reads it at the published layer in 17 of 19 process runs, but a one-gap push of the oracle never moves it, while for the two controls it always does. In exploratory pushes up to eight gaps, targets that are a choice of answer become pushable along the oracle, while targets that describe the course of the computation (settling depth, chain-of-thought necessity, backtracking) move in one of twelve runs, and the published directions almost never move their targets. Two (known-entity awareness, reasoning strength) fail to reproduce their authors' headline effect in at least two models; where reasoning strength does reproduce, it is a length direction. Behind these verdicts, known-entity awareness and true thinking track their processes in every panel model and beat both label-free competitors, a reasoning direction again moves reply format rather than computation, and the sycophancy vector reads as agreement with the user rather than as giving in. Weight-randomised models reproduce calibration separation at AUC ≥ 0.80 for four of nine directions. Effects judged against norm-matched random directions exceed effects judged against KL-matched ones by a quarter of a smallest effect of interest on average. Files. direction-validity-battery.pdf (the paper); direction-validity-battery-source.zip (the repository: preregistrations with their SHA-256 records, the deviations log, all code, item pools, per-run summaries, the cross-direction aggregate, the exploratory strength analyses, the results page and the LaTeX source); results-per-item-part01.zip to part10.zip (every main-run per-item record, split into ten parts to keep each under 10 MB; unzip all into one folder; all model-written text removed and no direction vectors). Code: github.com/Somebodyhere101/direction-validity-battery. Use of AI assistance. The idea of packaging the series' checks as a battery and applying it to published directions is the author's. An AI system (Claude, Anthropic), working under the author's direction, reviewed the literature, wrote the preregistrations and code, ran the experiments and analyses, and drafted the paper. The author directed the work and is responsible for its content.
Author: Ramdan, Sam
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7222895291