A sharp factor d/2 for diagonal preconditioning of single-sample SGD
Year: 2026
Type: preprint
Abstract: We study single-sample stochastic gradient descent (SGD) with a constant step on noiseless Gaussian least squares with feature covariance Σ, preconditioned by a positive diagonal matrix P. Let γ* be the exact asymptotic decay rate of the expected loss at the best constant step, and let R(Σ) be the ratio of the rates of the two Gauss–Newton diagonal preconditioners diag(Σ)^{-1/2} and diag(Σ)^{-1}. Main theorem. For every dimension d ≥ 3 and every Σ, R(Σ) < d/2, and sup_Σ R(Σ) = d/2; the supremum is not attained. Equivalently, no diagonal preconditioner improves the exact rate of diag(Σ)^{-1} (Jacobi scaling) by a factor d/2 or more, and the factor d/2 is sharp. This contrasts with the spectral condition number, where the worst-case factor of Jacobi scaling equals n/2 only for 2 ≤ n ≤ 5. Further results. Elliptical features. Through an exact operator rescaling with effective batch size n_{b,eff} = 1 + (n_b − 1)/κ_ell, the theorem and its quantitative form extend literally to every elliptical feature distribution with finite fourth moment. Dimension-independent companion bound. R(Σ) ≤ κ* + 2κ*(κ* − 1)/(d + 2), where κ* is the optimal diagonally scaled condition number. Sharp quantitative form. For d ≥ 60 we prove, without interval arithmetic, R(Σ) < (d/2)(1 − min((1/40)√(d λ_min(C)), 0.0029)). This matches the lower-bound family R > (d/2)(1 − (5/2)√(d λ_min(C))), with double-limit constant (4/3)√(d λ_min(C)) as λ_min(C) → 0 and d → ∞, and closes the relative gap at the exact joint scaling Θ(√(d λ_min(C))). Mini-batches. The factor d/2 fails for mini-batches already at n_b = 5 (for d ≥ 85) and explicitly at n_b ≥ 135 for d = 13. The Householder angle θ = π is optimal in our explicit involution family. Two-block Jacobi. Two-block Jacobi preconditioning in single-sample SGD has a linear breakdown ≥ (b+2)/3 = (d+4)/6, despite being exactly optimal for the full-batch condition number. Proof structure and verification. The proof is computer-assisted only in dimensions 3 ≤ d ≤ 14, using interval arithmetic (40-digit outward rounding, exact volume ledgers) confined to the compact remainder. Every dimension d ≥ 15 (both the outer and inner regimes), as well as the fourth-eigenvalue floor and its collective sum-floor companion for all d ≥ 4, are proved by hand up to one-variable rational identities and Descartes sign checks in exact rational arithmetic. Relevance: it quantifies how much a diagonal preconditioner (Adam-like, Jacobi, Gauss–Newton diagonal) can possibly gain over plain Jacobi scaling for SGD.
Author: Paiva, Gabriel Costa de Mello
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7235017813
