Year: 2025
Type: preprint
Abstract: Stochastic gradient optimization is the dominant learning paradigm for a variety of scenarios, from classical supervised learning to modern self-supervised learning. We consider stochastic gradient algorithms for learning problems whose objectives rely on unknown nuisance parameters, and establish non-asymptotic convergence guarantees. Our results show that, while the presence of a nuisance can alter the optimum and upset the optimization trajectory, the classical stochastic gradient algorithm may still converge under appropriate conditions, such as Neyman orthogonality. Moreover, even when Neyman orthogonality is not satisfied, we show that an algorithm variant with approximately orthogonalized updates (with an approximately orthogonalized gradient oracle) may achieve similar convergence rates. Examples from orthogonal statistical learning/double machine learning and causal inference are discussed.
Source: arXiv (Cornell University)
Institution: University of Washington
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: Collaborative Research: MFAI: Mathematical Frontiers of Generative AI, G4243306591, G4321346201, TRIPODS: Institute for Foundations of Data Science, AI Institute: Institute for Foundations of Machine Learning, Numerical Construction of Optimal Estimators Using Machine Learning Tools, Scaling Laws of Deep Learning
OpenAlex ID: W7079199537
Cites:
Cited by:
Related to: