Year: 2026
Type: preprint
Abstract: Whether a deep residual architecture avoids numerical blow-up in its forward and backward dynamics is typically determined through costly trial-and-error training. Existing stabilization strategies, including normalization and residual scaling, lack a common framework for predicting their effects across architectures. We develop an a priori certification framework based on two exponents derived from architectural primitives: the forward growth exponent $q$ and the Jacobian sensitivity exponent $σ$. These exponents characterize stability regimes and determine how certified allowable depth scales with the floating-point limit $M$. In particular, $q=1$ marks a sharp threshold: for $σ>0$, certified depth scales as $\log M/\log\log M$, $\sqrt{\log M}$, and $\log_q\log M$ in the regimes $q<1$, $q=1$, and $q>1$, respectively, exhibiting changes reminiscent of phase transitions in statistical physics. The sensitivity exponent $σ$ further constrains backward stability, even when $q\le1$. Exponent arithmetic enables certification from computation graphs, including heterogeneous residual stacks, and identifies targeted modifications to existing backbones without uniform normalization. Experiments on Mamba, PatchTST, FNO, GPT-2, and a Toto~2.0 proxy support the predicted stability distinctions: controlling superlinear growth prevents divergence in Mamba and Toto~2.0, while reducing attention sensitivity restores stable GPT-2 training. Among stable designs, $q=1$ architectures can outperform normalized $q=0$ counterparts in forecasting and operator-learning extrapolation. Rather than prescribing a single architecture, our framework identifies families of architectures with certifiable stability, narrowing the scope of trial-and-error and enabling systematic, task-dependent exploration before training.
Source: arXiv (Cornell University)
Institutions: SRI International, University of Massachusetts Amherst
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7169542570