Work

Complexity and performance for two classes of noise-tolerant first-order algorithms

Year: 2025

Type: article

Abstract: Two classes of algorithms for optimization in the presence of noise are presented, that do not require the evaluation of the objective function. The first generalizes the well- known Adagrad method. Its complexity is then analyzed as a function of its parameters. A second class of algorithms is then derived whose complexity is at least as good as that of the first class. Initial numerical experiments on finite-sum problems arising from deep-learning applications suggest that methods of the second class may outperform those of the first.

Source: Optimization methods & software

DOI: 10.1080/10556788.2025.2532736

Authors: S. Gratton, S. Jerad, Ph. L. Toint

Institutions: Institut National Polytechnique de Toulouse, Institut de Recherche en Informatique de Toulouse, Université de Toulouse, University of Namur, Namur Institute for Complex Systems

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: Agence Nationale de la Recherche, Centre National de la Recherche Scientifique, European Regional Development Fund, Artificial and Natural Intelligence Toulouse Institute

Awards: G3010527862, G3879958799, G602161992