Work

A Continuized View on Nesterov Acceleration

Year: 2021

Type: preprint

Abstract: We introduce the "continuized" Nesterov acceleration, a close variant of Nesterov acceleration whose variables are indexed by a continuous time parameter. The two variables continuously mix following a linear ordinary differential equation and take gradient steps at random times. This continuized variant benefits from the best of the continuous and the discrete frameworks: as a continuous process, one can use differential calculus to analyze convergence and obtain analytical expressions for the parameters; but a discretization of the continuized process can be computed exactly with convergence rates similar to those of Nesterov original acceleration. We show that the discretization has the same structure as Nesterov acceleration, but with random parameters.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2102.06035

Authors: Berthier, Raphaël, Bach, Francis, Flammarion, Nicolas, Gaillard, Pierre, Taylor, Adrien

Institutions: Université Paris Sciences et Lettres, University of Georgia, Université Grenoble Alpes, THOTH: Apprentissage de modèles visuels à partir de données massives

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funder: Agence Nationale de la Recherche

OpenAlex ID: W3127779101

Cites: 19

Cited by:

Related to: 10