Does your readout see the input? A sanity check for linear probing of representations
Year: 2026
Type: preprint
Abstract: To ask what a representation contains, researchers usually fit a linear readout: a regression or classifier trained on it. That readout is a model with inputs of its own. It can be given the representation alone, or together with the original features: two different questions, two different answers. Given both, its score stops measuring the representation and starts measuring the input. On a public fMRI dataset the arithmetic reproduces to the last digit, and the same design raises reported decoding accuracy by 22% to 326%. I show that this is not a bias of unknown size but a quantity that can be computed from two scores. The apparent advantage of supplying the features equals C/B − 1, where B is the score of a readout fitted on the representation alone and C the score of one fitted on the features alone. In a reconstruction probe the features-only readout became numerically indistinguishable from a readout given the full representation, so the metric lost its ability to rank representations at all. With a classifier, the same setting inflated accuracy by tens of percent. The effect tracked how much of the input the representation retained and did not depend on how the representation was learned: a trained autoencoder reproduced the result of PCA, and a second dataset reproduced it again. I give a seven-item checklist and a one-function diagnostic built only on scikit-learn, and I report the predictions I wrote down before running, including the two that my data refuted.
Author: Jiang, Yuntong
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7226007202
