Get started
Work

Exact Second-Moment Dynamics, Critical Batch Scaling, Vocabulary Phase Transitions, and the Feshbach–Guyan–Southwell Counterdiabatic Replacement for Adam

Year: 2026
Type: preprint
Abstract: For heavy-ball SGD on a quadratic loss in R^P with minibatch Hessians H_{B_t} of mean H ≻ 0 at finite dimension P and finite batch size B, we analyze the exact second-moment dynamics when features have independent coordinates in the eigenbasis of H with mode-dependent fourth moments 3 + κ_k and a common random scale of fourth moment γ ≥ 1. Although mode-dependent kurtoses place the curvature noise outside the spectrally expressible class, the second-moment operator T has the spectral radius of its restriction to diagonal blocks at every step size η, the stability threshold is the root of a rank-one secular equation S(η) = 1, the expected loss obeys an exact discrete convolution equation (a renewal equation when γ(2 + κ_k) ≥ 1), and ρ(T) = 1/z⋆ where z⋆ is the smallest positive root of z K̂(z) = 1. On the stable side with γ(2 + κ_k) ≥ 1, the loss approaches its floor as a single exponential C ρ(T)^t with an explicit C > 0, and 1 − ρ(T) vanishes linearly at the threshold with an explicit renewal slope. Along any fixed noise floor S(η) = s ∈ (0, 1), we prove that (1 − ρ(T))/(1 − β) → 1 − η_EMA(1; s)/η⋆_EMA(1) as β ↑ 1 with an explicit O(1 − β) correction, and characterize when anisotropic spectra yield a momentum speedup strictly above 2. Applying the theory to the vocabulary model of Everett and Paquette (2026), we settle their open problems at finite (d, B, η, β) and extend the theory to construct the Feshbach–Guyan–Southwell (FGS) Counterdiabatic Optimizer, a provably optimal replacement for Adam (Kingma and Ba, 2015): (i) we solve the exact minimax envelope W_θ(B̆) for all power-law preconditioners η_j = α q_j^(−θ) (θ ∈ [0, 1], including Adam at θ = 1/2), proving the Momentum–Preconditioning Duality S_{θ,mom}(∞) ≡ S_{(1+θ)/2,SGD}(∞), showing that Adam delays beneficial momentum to B×(1/2) = 2(√κ_V + 1) B×(0) > 4 B×(0), and proving that within the damped family α(q_j + ε²)^(−θ) only θ = 1 with mobility-edge damping ε⋆²(B) = (d+2)/(B−1) and β = 0 achieves the Mandelstam–Tamm speed limit across V ≥ 3 tokens; (ii) via an exact Feshbach–Poincaré stroboscopic return map at active token arrival times, we prove that β = 0 (Guyan static condensation) contracts rare rows by (d+1)/(d+2) per hit with zero silent-step memory traffic, whereas Adam's finite-β₂ accumulator collapses to a non-contracting SignSGD floor when p_q ≪ 1 − β₂; and (iii) under label noise σ² > 0 and anisotropic embeddings H_j = q_j H̃_j, Equal-Rate FGS water-filling with co-scaled critical damping 1 − β⋆_j = 4 q_j η_{EMA,j} and HVP-free online Southwell trace estimation reduces sample complexity by 1/(V q_min) over global Heavy Ball and 2/(V q_min) over global SGD. This is the full, extended version (54 pages) of the author's earlier preprint "Minibatch Heavy-Ball SGD at Finite Dimension: Exact Threshold, Rate and Critical Slowing Down" (Zenodo, 5 November 2026, https://zenodo.org/records/23972672). Beyond the results of that note, it adds the vocabulary-model theory (critical batch size, bifurcation and bend scale, label-noise phase transition, anisotropic token embeddings), the FGS and Universal FGS (UFGS) counterdiabatic optimizers as replacements for Adam and AdamW, extensions to logistic and softmax cross-entropy losses, and implementable optimum-free, HVP-free estimators. Preliminary release, not peer reviewed. Results and proofs remain subject to independent mathematical verification. Claims of general superiority over practical AdamW or Muon should not be inferred from guarantees proved for restricted quadratic or predictable-preconditioner models. This public draft documents the ideas and their development; it is not a certificate of theorem correctness, novelty, or conference acceptance.
DOI: 10.5281/zenodo.24074235
Language: English
Open Access status: green
OpenAlex ID: W7225640385
Cites:
Cited by:
Related to: