Year: 2021
Type: article
Abstract: Stochastic optimization methods have become a class of popular optimization tools in machine learning. Especially, stochastic gradient descent (SGD) has been widely used for machine learning problems, such as training neural networks, due to low per-iteration computational complexity. In fact, the Newton or quasi-newton (QN) methods leveraging the second-order information are able to achieve a better solution than the first-order methods. Thus, stochastic QN (SQN) methods have been developed to achieve a better solution efficiently than the stochastic first-order methods by utilizing approximate second-order information. However, the existing SQN methods still do not reach the best known stochastic first-order oracle (SFO) complexity. To fill this gap, we propose a novel faster stochastic QN method (SpiderSQN) based on the variance reduced technique of SIPDER. We prove that our SpiderSQN method reaches the best known SFO complexity of$\mathcal {O}(n+n^{1/2}\epsilon ^{-2})$in the finite-sum setting to obtain an$\epsilon $-first-order stationary point. To further improve its practical performance, we incorporate SpiderSQN with different momentum schemes. Moreover, the proposed algorithms are generalized to the online setting, and the corresponding SFO complexity of$\mathcal {O}(\epsilon ^{-3})$is developed, which also matches the existing best result. Extensive experiments on benchmark data sets demonstrate that our new algorithms outperform state-of-the-art approaches for nonconvex optimization.
Authors: Qingsong Zhang, Feihu Huang, Cheng Deng, Heng Huang
Institutions: Xidian University, Jingdong (China), University of Pittsburgh
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: SCH: EXP: Collaborative Research: Privacy-Preserving Framework for Publishing Electronic Healthcare Records, G2241343584, BIGDATA: Collaborative Research: IA: Big Imaging-Omics Data Mining Framework for Precision Medicine, SCH: INT: New Machine Learning Framework to Conduct Anesthesia Risk Stratification and Decision Support for Precision Health, G4470460840, 基于视觉语言驱动的多模态认知方法研究, III: Medium: Collaborative Research: Robust Large-Scale Electronic Medical Record Data Mining Framework to Conduct Risk Stratification for Personalized Intervention, III: Small: Robust Large-Scale Data Mining for Knowledge Discovery in Depression Thought Records, BIGDATA: IA: Collaborative Research: Asynchronous Distributed Machine Learning Framework for Multi-Site Collaborative Brain Big Data Mining
OpenAlex ID: W3134948707