How Residual-Stream Updates Travel Through Depth: A Pre-Registered Study of Cross-Layer Dynamics in Transformer Factual Recall
Year: 2026
Type: preprint
Abstract: Transformer components communicate through an additive residual stream, so the update a layer writes, Δ_ℓ = v_{ℓ+1} − v_ℓ, is the natural unit of cross-layer computation. We pre-registered two hypotheses about these updates on factual-recall inputs — (H1) they are predictable across depth; (H2) the prediction kernel is approximately depth-stationary — and ran the frozen protocol on Pythia-410m-deduped, replicating on five further models from three families. (1) H1 holds far above pre-registered nulls but decomposes exactly: an orthogonal split of the held-out explained variance attributes most of it to the running-sum tautology on the four deepest updates (0.65 = 0.52 + 0.13) and roughly half on the whole stack (raw 0.52 = 0.27 + 0.25), and on the primary model what survives is substantially architectural — an untrained network exceeds it (though on GPT-2-small it is about half learned). (2) H2 fails decisively: kernels fit on one depth band predict another band's updates worse than that band's mean, so the right abstraction is a first-order (k* ≈ 1), fully depth-indexed state-space model (SSM), validating the unconstrained parameterisation of cross-layer transcoders. (3) Fitted over frozen sparse sources, the SSM reconstructs Δ_ℓ above a matched per-layer dictionary (+0.049 R² rank-matched, +0.008 parameter-matched), with a depth-graded, roughly half-learned gain; its gauge-invariant state path attenuates by half per layer of depth gap (memory ≈ 1.2 layers, consistent with k* ≈ 1) and the state decodes the recalled answer at mid-depth better than dimension-matched PCA of its own codes. (4) The pre-registered causal test returns a controlled null: no single feature's ablation clears threshold, and the group effect that does is reproduced by a norm-matched random direction — recall is a distributed, magnitude-driven code. As a by-product, on the primary model TopK dictionaries make single-feature directions seed-unidentifiable where matched L1 dictionaries on the same activations are perfectly stable. All gates were frozen before data collection, every headline reconstruction gap carries a paired bootstrap confidence interval, and every figure regenerates from tracked tables. Code, frozen protocol, figures and run records: https://github.com/aljazfrancic/delta-stream
Author: Frančič, Aljaž
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225452543