Year: 2022
Type: preprint
Abstract: Fueled by advances in distributed deep learning (DDL), recent years have witnessed a rapidly growing demand for resource-intensive distributed/parallel computing to process DDL computing jobs. To resolve network communication bottleneck and load balancing issues in distributed computing, the so-called ``ring-all-reduce'' decentralized architecture has been increasingly adopted to remove the need for dedicated parameter servers. To date, however, there remains a lack of theoretical understanding on how to design resource optimization algorithms for efficiently scheduling ring-all-reduce DDL jobs in computing clusters. This motivates us to fill this gap by proposing a series of new resource scheduling designs for ring-all-reduce DDL jobs. Our contributions in this paper are three-fold: i) We propose a new resource scheduling analytical model for ring-all-reduce deep learning, which covers a wide range of objectives in DDL performance optimization (e.g., excessive training avoidance, energy efficiency, fairness); ii) Based on the proposed performance analytical model, we develop an efficient resource scheduling algorithm called GADGET (greedy ring-all-reduce distributed graph embedding technique), which enjoys a provable strong performance guarantee; iii) We conduct extensive trace-driven experiments to demonstrate the effectiveness of the GADGET approach and its superiority over the state of the art.
Source: arXiv (Cornell University)
Authors: Yu, Menglu, Tian, Ye, Ji, Bo, Wu, Chuan, Rajan, Hridesh, Liu, Jia
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: CAREER: Theory and Algorithms for Efficient Control of Wireless Networks with Jointly Optimized Performance: High Throughput, Low Delay, and Low Complexity, AI Institute for Future Edge Networks and Distributed Intelligence (AI-EDGE), CAREER: Computing-Aware Network Optimization for Efficient Distributed Data Analytics at the Wireless Edge, Collaborative Research: CCRI: ENS: Boa 2.0: Enhancing Infrastructure for Studying Software and its Evolution at a Large Scale, NeTS: Small: Toward Optimal, Efficient, and Holistic Networking Design for Massive-MIMO Wireless Networks, SpecEES: Toward Spectral and Energy Efficient Cross-Layer Designs for Millimeter-Wave-Based Massive MIMO Networks, HDR TRIPODS: D4 (Dependable Data-Driven Discovery) Institute, CIF: Small: Taming Convergence and Delay in Stochastic Network Optimization with Hessian Information, G6671297155
OpenAlex ID: W4287929077