UniVR: A Universal Variance Reduction Framework for Proximal Stochastic Gradient Method.
Year: 2015
Type: preprint
Abstract: We revisit an important class of composite stochastic minimization problems that often arises from empirical risk minimization settings, such as Lasso, Ridge Regression, and Logistic Regression.
We present a new algorithm UniVR based on stochastic gradient descent with variance reduction. Our algorithm supports non-strongly convex objectives directly, and outperforms all of the state-of-the-art algorithms, including both direct algorithms (SAG, MISO, and SAGA) and indirect algorithms (SVRG, ProxSVRG, SDCA, ProxSDCA, and Finito) for such objectives. Our algorithm supports strongly convex objectives as well, and matches the best known linear convergence rate. Experiments support our theory.
As a result, UniVR closes an interesting gap in the literature because all the existing direct algorithms for the non-strongly convex case perform much slower than the indirect algorithms. We thus believe that UniVR provides a unification between the strongly and the non-strongly convex stochastic minimization theories.
Source: arXiv (Cornell University)
Authors: Zeyuan Allen Zhu, Yuan Yang
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W1553774930
