Year: 2023
Type: conference-paper
Abstract: Many machine-learning applications rely on distributed machine learning (DML) systems to train models from massive datasets using massive computing resources (e.g., GPUs and TPUs). However, given a DML system in most applications, its parameter synchronization mechanism is fixed and independent from the types and amounts of resources available to the system. In this paper, we argue that given an application, the synchronization mechanism in its DML system should be co-designed with the available resources in the heterogeneous cluster. To this end, we design an efficient parameter synchronization framework called Baileys. First, Baileys retrieves resource information from the heterogeneous clusters and uses such information to partition heterogeneous workers into multiple groups dynamically. Second, Baileys develops efficient group-based parameter synchronization mechanisms to converge model parameters quickly and accurately. We implement a prototype of Baileys and demonstrate its efficiency and efficacy through experiments. Results show that Baileys can reduce block time up to 33.2% and decrease the test error up to 39.7%.
Authors: Chengdong Ni, Haizhou Du
Institution: Shanghai University of Electric Power
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W4386522657