Work

Adaptive Spectral Regularization for Long-Horizon Reasoning in Autoregressive Transformers

Year: 2026
Type: article
Abstract: This paper investigates why autoregressive transformers suffer from compounding reasoning errors and state-tracking failures over extended generation horizons. By modeling multi-step generation as a discrete-time dynamical system, the paper proves that unregularized singular values in attention and feed-forward projections amplify early-step representation perturbations exponentially with reasoning depth. To solve this without restricting model expressivity on deterministic tasks, the paper introduces Adaptive Spectral Regularization (ASR)—a differentiable training constraint that dynamically tightens the upper bound on projection matrix spectral norms when next-token predictive entropy is high, while relaxing the constraint during low-entropy steps. The work provides a formal upper bound on cumulative horizon drift, derives an O(dindout)O(dindout) single-step power-iteration estimator for efficient training, and establishes a controlled benchmark protocol across symbolic deduction, algorithmic state-tracking, and long-context retrieval.
Author: Akash
Language: English
Open Access status: green
OpenAlex ID: W7225920483
FWCI:
Cites:
Cited by:
Related to: 10