What Does Adam Bias Correction Actually Do? Exact Moment Identities, the Step-Size Multiplier sqrt(1-beta2^t)/(1-beta1^t), and Full-Batch Tests on Six UCI Regression Designs
Year: 2026
Type: preprint
Abstract: Adam maintains exponential averages of gradients and squared gradients and divides by 1-β^t to unbias them. I prove that if gradients are iid with mean μ and variance σ^2, then E[m_t]=(1-β1^t)μ and E[v_t]=(1-β2^t)(μ^2+σ^2) exactly, so the bias-corrected ratio has expectation near μ/sqrt(μ^2+σ^2) while the uncorrected ratio equals that times (1-β1^t)/sqrt(1-β2^t). With default β1=0.9, β2=0.999 the uncorrected step is larger early on: about 3.16 times at t=1 and 6.53 times at t=10. On six centred and standardised UCI regression problems with full-batch Adam at step size 0.05, removing bias correction reached a 1% excess-loss threshold in fewer iterations on every data set (for example 188 versus 500 on the power-plant set, 52 versus 360 on yacht). Bias correction therefore shrinks early Adam steps relative to the raw ratio m/sqrt(v); whether that helps depends on the step-size choice.
Author: Aguru, Manoj Kumar
Institution: Chaitanya
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225454464