Work

When Probes Succeed for the Wrong Reason: Causal Regimes of Probe-Identified Representations

Year: 2026
Type: preprint
Abstract: Linear probes are widely used in LLM interpretability, but does high accuracy mean the probe found something causally important? We test this by training probes on refusal behavior in two models, Phi-3 (3.8B) and Mistral-7B, and then using activation interventions to check whether the probe directions are actually necessary. The answer depends on the model. In Phi-3, refusal lives in a low-rank subspace: removing it makes the behavior collapse. In Mistral-7B, refusal is distributed across many redundant directions: adding them steers behavior, but removing them does nothing. High probe accuracy does not tell you which regime you are in.
Language: English
Open Access status: green
OpenAlex ID: W7234598661
Cites:
Cited by:
Related to: 10