Work

A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets

Year: 2012

Type: preprint

Abstract: We propose a new stochastic gradient method for optimizing the sum of a finite set of smooth functions, where the sum is strongly convex. While standard stochastic gradient methods converge at sublinear rates for this problem, the proposed method incorporates a memory of previous gradient values in order to achieve a linear convergence rate. In a machine learning context, numerical experiments indicate that the new algorithm can dramatically outperform standard algorithms, both in terms of optimizing the training error and reducing the test error quickly.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.1202.6258

Authors: Roux, Nicolas Le, Schmidt, Mark, Bach, Francis

Institutions: Institut national de recherche en sciences et technologies du numérique, École Normale Supérieure - PSL, Département d'Informatique, SIERRA: Apprentissage Statistique et optimisation

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funder: Natural Sciences and Engineering Research Council of Canada

OpenAlex ID: W2105875671

Cites: 40

Cited by: 577

Related to: 10