Do a Coding Model's Activations Warn of Test-Gaming Earlier than Its Text? A Pre-registered Test
Year: 2026
Type: preprint
Abstract: When a coding agent is about to game its tests, does reading its internal activations tell you before reading its text does? I test this on HackTrace, a dataset of 173k Qwen3-8B coding attempts labelled as cheating or honest, with a pre-registered design: detectors, thresholds and decision rules were fixed before any probe was scored, the main claim was tested on a development split, and then confirmed once on a reserved set of problems that had never been opened. The answer is no. A linear probe reproduces HackTrace's in-distribution result (within-problem AUROC 0.997), but in a token-by-token replay of complete attempts at one false alarm per 20 honest attempts, it warns before the first shortcut token on 10.4% of cheats on the development split and 3.4% on the reserved set, against 11.0% and 5.6% for the best text monitor. Stronger readouts, smoothing, a monitor that reads the code written so far, and a comparison at matched false-alarm rates do not change this; a frontier judge reading each finished turn catches 92-95% of cheats before submission, more than any probe. The probe's robustness to disguised cheats (0.998 while TF-IDF falls to 0.86) is shared by a frontier judge (0.997) and by a probe on a different model's activations (0.990), so it is not specific to the model that wrote the code. On a second model family, Gemma-4-12B-it, the probe is below text monitors (0.76 vs 0.82; as a monitor, 23% vs 79% of cheats caught). Open models up to 27B almost never cheat unprompted on ImpossibleBench. For this kind of shortcut, the writing model's activations carried no early warning beyond its text that I could measure in time to act on.
Author: Puranik, Yashaswin Sharma
Language: English
Subfield: Education
Field: Social Sciences
Domain: Social Sciences
Open Access status: green
OpenAlex ID: W7234034783
