Work

Distributed Machine Learning

Year: 2024

Type: conference-paper

Abstract: We explore the landscape of distributed machine learning, focusing on advancements, challenges, and potential future directions in this rapidly evolving field. We delve into the motivation for distributed machine learning, its essential techniques, real-world applications, and open research questions. The theoretical discussion will give an overview of proving the convergence of popular Stochastic Gradient Descent (SGD) Algorithms to train contemporary machine learning models, including the deep learning models with the assumption of non-convexity, in a distributed setting. We will specify the convergence of data parallel SGD for various distributed systems properties, such as asynchronous and compressed communication. We will also discuss distributed machine learning techniques such as model parallelism and tensor parallelism to train large language models (LLMs).

Source: International Conference on Distributed Computing and Networking (ICDCN)

DOI: 10.1145/3631461.3632516

Author: Bapi Chatterjee

Institution: Indraprastha Institute of Information Technology Delhi

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: bronze

Funder: Universitas Brawijaya

OpenAlex ID: W4391090502

FWCI: 1.069

Cites: 6

Cited by: 7

Related to: 10