Work

Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression

Year: 2024

Type: conference-paper

Abstract: Gradient aggregation has long been identified as a major bottleneck in today's large-scale distributed machine learning training systems. One promising solution to mitigate such bottlenecks is gradient compression, directly reducing communicated gradient data volume. However, in practice, many gradient compression schemes do not achieve acceleration of the training process while also preserving accuracy.

Source: ACM Workshop on Hot Topics in Networks (HotNets)

DOI: 10.1145/3696348.3696857

Authors: Wenchen Han, Shay Vargaftik, Michael Mitzenmacher, Brad Karp, Ran Ben Basat

Institutions: University College London, VMware (United States), Harvard University

Language: English

Topic: Stochastic Gradient Optimization Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funder: National Science Foundation

Awards: Foundations of Data Science Institute, AF: Small: Algorithms and Data Structures with Predictions, CNS Core: Medium: Approximation and Randomization in the Programmable Data Plane