Quantized Frank-Wolfe: Communication-Efficient Distributed Optimization.
Year: 2019
Type: preprint
Abstract: How can we efficiently mitigate the overhead of gradient communications indistributed optimization? This problem is at the heart of training scalable
machine learning models and has been mainly studied in the unconstrained
setting. In this paper, we propose Quantized Frank-Wolfe (QFW), the first
projection-free and communication-efficient algorithm for solving constrained
optimization problems at scale. We consider both convex and non-convex
objective functions, expressed as a finite-sum or more generally a stochastic
optimization problem, and provide strong theoretical guarantees on the
convergence rate of QFW. This is done by proposing quantization schemes that
efficiently compress gradients while controlling the variance introduced during
this process. Finally, we empirically validate the efficiency of QFW in terms
of communication and the quality of returned solution against natural
baselines.
Source: arXiv (Cornell University)
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2913116962