Work

Convergence of adaptive algorithms for weakly convex constrained optimization

Year: 2020

Type: preprint

Abstract: We analyze the adaptive first order algorithm AMSGrad, for solving a constrained stochastic optimization problem with a weakly convex objective. We prove the $\mathcal{\tilde O}(t^{-1/4})$ rate of convergence for the norm of the gradient of Moreau envelope, which is the standard stationarity measure for this class of problems. It matches the known rates that adaptive algorithms enjoy for the specific case of unconstrained smooth stochastic optimization. Our analysis works with mini-batch size of $1$, constant first and second order moment parameters, and possibly unbounded optimization domains. Finally, we illustrate the applications and extensions of our results to specific problems and algorithms.

Source: arXiv (Cornell University)

DOI: 10.48550/arxiv.2006.06650

Authors: Alacaoglu, Ahmet, Malitsky, Yura, Cevher, Volkan

Institution: École Polytechnique Fédérale de Lausanne

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: National Science Foundation, European Commission, Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung, Hasler Stiftung, U.S. Navy, Office of Naval Research

Awards: G1886453495, G4254929486, G8392782456

OpenAlex ID: W3034845664

Cites: 21

Cited by: 4

Related to: 20