When Probes Succeed for the Wrong Reason: Causal Regimes of Probe-Identified Representations
Year: 2026
Type: preprint
Abstract: Linear probes are widely used in LLM interpretability, but does high accuracy mean the probe found something causally important? We test this by training probes on refusal behavior in two models, Phi-3 (3.8B) and Mistral-7B, and then using activation interventions to check whether the probe directions are actually necessary. The answer depends on the model. In Phi-3, refusal lives in a low-rank subspace: removing it makes the behavior collapse. In Mistral-7B, refusal is distributed across many redundant directions: adding them steers behavior, but removing them does nothing. High probe accuracy does not tell you which regime you are in.
Authors: MS, NEHAA SHRI, Warrier, Maitreya Prasad
Institution: Vellore Institute of Technology University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7234598661
