Rotation or Correctness? A Confound Control for Trajectory-Geometry Studies of LLM Self-Refinement
Year: 2026
Type: preprint
Abstract: A methods contribution for interpretability research on large language model self-refinement. Studies that correlate hidden-state trajectory geometry (rotation, curvature, displacement) with a model's error signal typically pool across all problems. This paper shows that such pooled correlations conflate two distinct effects: a refinement effect — geometry reflecting the model actually revising its answer — and a correctness effect — geometry that merely differs between right and wrong answers, independent of any revision. It introduces a simple control: stratify problems by whether the answer changed between an initial and a final response, and recompute the correlation within the refinement-active and stable strata separately. Applied across four model families (Qwen2.5-7B, Llama-3-8B, Mistral-7B-v0.3, Gemma-2-9B) and three task types (GSM8K, MATH, HumanEval) — 1,650 instrumented problems — the result is uniform and negative: no refinement-specific geometric signal survives the control. The contribution is the control itself, released as a reusable diagnostic, together with the negative result that demonstrates it catches a real confound. Code: https://github.com/gvillines-hub/refinement-confound-control
Author: Villines, Gregory
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7162014646
Cites:
Cited by:
Related to: