Work

Restart and Adaptive Acceleration in Stochastic Gradient Methods

Year: 2026

Type: preprint

Abstract: We study restart schemes in stochastic optimization problems for non-smooth and weakly convex functions that satisfy a Kurdyka-Łojasiewicz (KŁ) inequality. Using restarts allows us to leverage KŁ inequalities to achieve optimal convergence rates, with acceleration depending explicitly on the KŁ exponent. Furthermore, for stochastic gradient descent (SGD), optimal restart schedules correspond to piecewise constant (step decay) step sizes. While regularity constants such as the KŁ exponent are typically unknown in practice, we prove that restart schemes are robust to significant misspecification of these constants, hence nearly adaptive. We detail numerical experiments on a number of problems where the KŁ exponent is controlled.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2606.21354

Authors: Elhishi, Ali, Roux, Chistophe, d'Aspremont, Alexandre

Institutions: Centre National de la Recherche Scientifique, Zuse Institute Berlin, École Normale Supérieure - PSL, Département d'Informatique, Technische Universität Berlin

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

OpenAlex ID: W7165674602

Cites: 5

Cited by: 1

Related to: 10