Year: 2026
Type: preprint
Abstract: In deep networks with small initialization, training can exhibit long plateaus separated by sharp feature-acquisition transitions. Whereas shallow nonlinear networks and deep linear networks are well studied, extending these analyses to deep nonlinear networks remains challenging. We derive exact scalar and matrix identities for the imbalance of layer weight norms, holding for any smooth activation and any differentiable loss, and use the resulting approximate balance law to control the full finite-width gradient flow through its first escape. This gives a scaling law in the escape time: with a prefix of $r$ layers initialized at the bottleneck scale $\varepsilon$, the escape time is $Θ(Γ^{-1})$ for $r=1$, $Θ(Γ^{-1}\log(1/\varepsilon))$ for $r=2$, and $Θ(Γ^{-1}\varepsilon^{-(r-2)})$ for $r\geq3$, where $Γ=\|\mathbb{E}[yx]\|$. We further show that a multimode teacher leaves the saddle once rather than in stages: with scalar output, all mode overlaps grow through a common amplitude, with relative weights fixed in advance. We find close agreement between our theory and experiment.
Source: arXiv (Cornell University)
Authors: Rawal, Divit, DeWeese, Michael R.
Institution: University of California, Berkeley
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G5259331294, G7452299184
OpenAlex ID: W7160311891