Depth-Locked Dynamics in Looped Language Models
Year: 2026
Type: preprint
Abstract: A looped transformer applies one block of layers many times. It is trained either at a fixed number of loops r or at a number of loops drawn at random for every sequence. We measure the dynamics inside the loop of 51 looped language models trained on the same data, and find that a loss placed at one fixed depth locks the dynamics to that depth. Before r, the loop takes Euler-like steps: each step shrinks as r grows, and loops of depth 6 to 24 trace the same step profile in r steps. At r, the loop brakes: its Jacobian contracts most within one loop of r in 28 of the 31 models trained at a fixed depth through every loop, including all 27 with r ≤ 16. Past r, the step keeps rotating in a plane. At r = 6 to 12 the state swings away and back and the loss rises and falls; one turn takes about two training depths. Scaling every update by r/r′ moves this landing to loop r′ = 1.5r or 2r without retraining, and keeps the downstream accuracy of loop r. In our controls, none of these signatures appears when the depth is random, even with backpropagation through every loop, or when the loss is placed at every loop. Between r and 2r the loss of all 12 such models rises by at most 0.003 nats, against 0.02 to 0.51 nats in all 31 fixed-depth models trained through every loop. The recipes also differ in contraction, which may mediate the effect. At matched contraction, the loss of a damped fixed-depth loop still rises by 0.19 to 0.23 nats from r to 2r, that of a damped random-depth loop by at most 0.015. We prove that a loss averaged over depths confines the state in the directions the readout uses. Every registered test had its threshold written before it was run; the few that followed a first look at the data are flagged.
Author: Cofone, Leonardo
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225988901
