Work

A Simplified Analysis of SGD for Linear Regression with Weight Averaging

Year: 2025

Type: preprint

Abstract: Theoretically understanding stochastic gradient descent (SGD) in overparameterized models has led to the development of several optimization algorithms that are widely used in practice today. Recent work by~\citet{zou2021benign} provides sharp rates for SGD optimization in linear regression using constant learning rate, both with and without tail iterate averaging, based on a bias-variance decomposition of the risk. In our work, we provide a simplified analysis recovering the same bias and variance bounds provided in~\citep{zou2021benign} based on simple linear algebra tools, bypassing the requirement to manipulate operators on positive semi-definite (PSD) matrices. We believe our work makes the analysis of SGD on linear regression very accessible and will be helpful in further analyzing mini-batching and learning rate scheduling, leading to improvements in the training of realistic models.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2506.15535

Authors: Meterez, Alexandru, Morwani, Depen, Oncescu, Costin-Andrei, Wu, Jingfeng, Pehlevan, Cengiz, Kakade, Sham

Institutions: Harvard University, University of California, Berkeley

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, Harvard University, Defense Advanced Research Projects Agency

Awards: A Theory of Learned Representations in Artificial and Natural Neural Networks, CAREER: Developing Neural Network Theory for Uncovering How the Brain Learns, G4523241894, G6671297155

OpenAlex ID: W4415334112

Cites:

Cited by:

Related to: