Representation Alignment and the Limits of Local Error Calibration for Multistep Tasks
Year: 2026
Type: preprint
Abstract: Two equivalent instructions can lead to different mistakes. We ask whether measuring exactly where a language model goes wrong helps predict success on new operation sequences or choose a better instruction. We use equivalent lookup tables and uninterrupted multistep responses from one Gemma configuration. In two independent development panels of 16 worlds each, local transition models had higher Brier loss than fixed candidate-by-length success rates. Selection gains over budget-capped direct search were 1.95 and 0.52 percentage points, below the prespecified 3-point target. In the second panel the full model and a simple summary selected identical materials in all 16 worlds. Matching the encoding of the initial state, rule table, and record improved multistep utility by 5.47 points in eight new worlds (80% development interval [2.73, 8.07]). Calibrating on uninterrupted trajectories yielded a local five-class Brier gain of 0.22005 over uniform error destinations, conditional on an observed legal error. The same calibration did not yield better whole-task forecasts or selections than matched controls. The results show that representation can materially affect execution even when local error models do not reliably identify the better representation before an answer. The planned formal evaluation was not run.
Author: TEO, EE HERN
Institution: Tianjin University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225331298