Review Residuals: Update-Conditioned Residual Gating for Transformers
Year: 2026
Type: preprint
Abstract: Residual connections add every sublayer's proposed update with a fixed coefficient of one; the network never evaluates whether an update is reliable before committing it. Drawing on the human-factors principle of independent verification — checking a proposed action before taking it — we introduce Review Residuals, which scale each update by a learned, input-dependent gate conditioned on both the current state and the proposed update. Conditioning the gate on the update is the property that distinguishes it from prior gated and scaled residuals. We report two findings. First, a depth-stability result: a convex (Highway-style) form of the gate reintroduces vanishing gradients and fails to train beyond ~20 layers, whereas the additive, identity-preserving form trains stably at all depths tested. Second, an emergence-with-scale result: trained from scratch across five sizes (60M–1B parameters, multi-seed), Review Residuals show no advantage at small scale but at 590M significantly outperform both a parameter-matched Highway gate and a parameter-matched standard residual (p<0.05), with a larger advantage at 1B. The benefit grows with model size rather than shrinking. Code, data, and figures are included with this record.
Author: Kramer, Kyle
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7166693845
Cites:
Cited by:
Related to: