Year: 2021
Type: article
Abstract: Distributed Stochastic Gradient Descent (SGD) when run in a synchronous manner, suffers from delays inruntimeas it waits for the slowest workers (stragglers). Asynchronous methods can alleviate stragglers, but cause gradient staleness that can adversely affect the convergenceerror. In this work, we present a novel theoretical characterization of the speedup offered by asynchronous methods by analyzing the trade-off between theerrorin the trained model and the actual trainingruntime(wallclock time). The main novelty in our work is that our runtime analysis considers random straggling delays, which helps us design and compare distributed SGD algorithms that strike a balance between straggling and staleness. We also provide a new error convergence analysis of asynchronous SGD variants without bounded or exponential delay assumptions. Finally, based on our theoretical characterization of the error-runtime trade-off, we propose a method of gradually varying synchronicity in distributed SGD and demonstrate its performance on the CIFAR10 dataset.
Authors: Sanghamitra Dutta, Jianyu Wang, Gauri Joshi
Institution: Carnegie Mellon University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funders: National Science Foundation, Qualcomm
OpenAlex ID: W3013308737
