Analytical Convergence Regions of Accelerated First-Order Methods in Nonconvex Optimization under Regularity Condition
Year: 2018
Type: preprint
Abstract: Gradient descent (GD) converges linearly to the global optimum for even nonconvex problems when the loss function satisfies certain benign geometric properties that are strictly weaker than strong convexity. One important property studied in the literature is the so-called Regularity Condition (RC). The RC property has been proven valid for many problems such as deep linear neural networks, shallow neural networks with nonlinear activations, phase retrieval, to name a few. Moreover, accelerated first-order methods (e.g. Nesterov's accelerated gradient and Heavy-ball) achieve great empirical success when the parameters are tuned properly but lack theoretical understandings in the nonconvex setting. In this paper, we use tools from robust control to analytically characterize the region of hyperparameters that ensure linear convergence of accelerated first-order methods under RC. Our results apply to all functions satisfying RC and therefore are more general than results derived for specific problem instances. We derive a set of Linear Matrix Inequalities (LMIs), based on which analytical regions of convergence are obtained by exploiting the Kalman-Yakubovich-Popov (KYP) lemma. Our work provides deeper understandings on the convergence behavior of accelerated first-order methods in nonconvex optimization.
Source: arXiv (Cornell University)
Authors: Huaqing Xiong, Yuejie Chi, Bin Hu, Wei Zhang
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2895073990