Communication-Efficient Frank-Wolfe Algorithm for Nonconvex Decentralized Distributed Learning
Year: 2021
Type: conference-paper
Abstract: Recently decentralized optimization attracts much attention in machine learning because it is more communication-efficient than the centralized fashion. Quantization is a promising method to reduce the communication cost via cutting down the budget of each single communication using the gradient compression. To further improve the communication efficiency, more recently, some quantized decentralized algorithms have been studied. However, the quantized decentralized algorithm for nonconvex constrained machine learning problems is still limited. Frank-Wolfe (a.k.a., conditional gradient or projection-free) method is very efficient to solve many constrained optimization tasks, such as low-rank or sparsity-constrained models training. In this paper, to fill the gap of decentralized quantized constrained optimization, we propose a novel communication-efficient Decentralized Quantized Stochastic Frank-Wolfe (DQSFW) algorithm for non-convex constrained learning models. We first design a new counterexample to show that the vanilla decentralized quantized stochastic Frank-Wolfe algorithm usually diverges. Thus, we propose DQSFW algorithm with the gradient tracking technique to guarantee the method will converge to the stationary point of non-convex optimization safely. In our theoretical analysis, we prove that to achieve the stationary point our DQSFW algorithm achieves the same gradient complexity as the standard stochastic Frank-Wolfe and centralized Frank-Wolfe algorithms, but has much less communication cost. Experiments on matrix completion and model compression applications demonstrate the efficiency of our new algorithm.
Authors: Wenhan Xian, Feihu Huang, Heng Huang
Institution: University of Pittsburgh
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
Awards: BIGDATA: Collaborative Research: IA: Big Imaging-Omics Data Mining Framework for Precision Medicine, SCH: INT: New Machine Learning Framework to Conduct Anesthesia Risk Stratification and Decision Support for Precision Health, III: Small: Robust Large-Scale Data Mining for Knowledge Discovery in Depression Thought Records, BIGDATA: IA: Collaborative Research: Asynchronous Distributed Machine Learning Framework for Multi-Site Collaborative Brain Big Data Mining, III: Medium: Collaborative Research: Integrating Large-Scale Machine Learning and Edge Computing for Collaborative Autonomous Vehicles, NSF Convergence Accelerator Track D: A Trusted Integrative Model and Data Sharing Platform for Accelerating AI-Driven Health Innovation
OpenAlex ID: W3176460985