Asymptotic Network Independence in Distributed Stochastic Optimization for Machine Learning: Examining Distributed and Centralized Stochastic Gradient Descent
Year: 2020
Type: article
Abstract: We provide a discussion of several recent results which, in certain scenarios, are able to overcome a barrier in distributed stochastic optimization for machine learning (ML). Our focus is the so-called asymptotic network independence property, which is achieved whenever a distributed method executed over a network of n nodes asymptotically converges to the optimal solution at a comparable rate to a centralized method with the same computational power as the entire network. We explain this property through an example involving the training of ML models and sketch a short mathematical analysis for comparing the performance of distributed stochastic gradient descent (DSGD) with centralized SGD.
Source: IEEE Signal Processing Magazine
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G2376368022, CPS: Breakthrough: A Dynamic Optimization Framework for Connected Automated Vehicles in Urban Environments, QuBBD: From Personalized Predictions to Better Control of Chronic Health Conditions, G3024551116, Optimizing and Learning Strategies for Protein Docking , SCH: INT: Distributed Analytics for Enhancing Fertility in Families, Efficiently Distributing Optimization over Large-Scale Networks, G8528306935
OpenAlex ID: W3022621540
