Adaptive Spectral Regularization for Long-Horizon Reasoning in Autoregressive Transformers
Year: 2026
Type: article
Abstract: This paper investigates why autoregressive transformers suffer from compounding reasoning errors and state-tracking failures over extended generation horizons. By modeling multi-step generation as a discrete-time dynamical system, the paper proves that unregularized singular values in attention and feed-forward projections amplify early-step representation perturbations exponentially with reasoning depth. To solve this without restricting model expressivity on deterministic tasks, the paper introduces Adaptive Spectral Regularization (ASR)—a differentiable training constraint that dynamically tightens the upper bound on projection matrix spectral norms when next-token predictive entropy is high, while relaxing the constraint during low-entropy steps. The work provides a formal upper bound on cumulative horizon drift, derives an O(dindout)O(dindout) single-step power-iteration estimator for efficient training, and establishes a controlled benchmark protocol across symbolic deduction, algorithmic state-tracking, and long-context retrieval.
Author: Akash
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225920483
