Work

Think and Answer Tokens Use Overlapping but Distinct Components of a Reasoning Model: Phase Restricted Ablations Against Token Group Baselines

Year: 2026
Type: preprint
Abstract: Reasoning models write a chain of thought before a final answer. Do the think and answer phases recruit different internal components? We mean ablate every attention head, attention layer and MLP of DeepSeek-R1-Distill-Qwen-1.5B on its own GSM8K traces and compare the importance profiles of think and answer tokens against their own measurement noise, with a disattenuated agreement statistic, bootstrap intervals, matched nulls and planted controls. Ablating each component only at the positions of the phase being scored, instead of everywhere, lowers head level agreement from 0.80 to 0.73 (95% CI 0.70 to 0.76). To give such values a scale we apply the same estimator to other pairs of token groups. Early and late think tokens agree at 0.91, while hard and easy think tokens agree at only 0.77, so in this model the phases differ about as much as easy and hard tokens do; matching think tokens to the answer tokens' difficulty does not raise agreement. Over the most important heads agreement is much lower (0.62 for the top 34). The answer share of importance rises with depth, but easy think tokens show a similar rise. Residual swaps with a displacement matched control confirm late layer answer specificity and trace negative think side effects to sharpened predictions. A MATH subset replicates these results. On Qwen3-1.7B restricted ablation again lowers agreement (0.81) and the top heads agree even less (0.38), but there the phases differ more than hard and easy tokens do and the depth trend survives difficulty matching. Its residual swaps show a different pattern. Random head sets show that the lower agreement of whole layers need not reflect layer structure. The phases share a backbone and differ measurably, most among the components that matter.
Language: English
Open Access status: green
OpenAlex ID: W7234338310
Cites:
Cited by:
Related to: 10