iPipe: An Efficient Adaptive Pipeline Parallel DNN Training System
Year: 2024
Type: conference-paper
Abstract: Pipeline parallelism is an efficient approach for distributed deep neural network (DNN) model training. However, existing pipeline parallel training systems often employ fixed model allocation strategies, leading to suboptimal efficiency in dynamic training scenarios. To address this issue, this paper proposes an efficient adaptive pipeline training system, iPipe, which dynamically manages the stages of the DNN model on GPUs based on their real-time states, effectively enhancing the training efficiency. In iPipe, a model allocation algorithm based on dynamic programming method is designed to search for an optimal model allocation plan with low complexity. Additionally, a live migration strategy which adopts an alternating computation method is proposed to achieve dynamic migration of model parameters between workers with minimal data transfer. We evaluate iPipe on a real GPU cluster, and the experimental results demonstrate that iPipe outperforms other pipeline parallel training systems in terms of overall training time for DNN models.
Authors: Qinghe Wang, Jian Yang, Xiaobin Tan, Mo Sha, Zhi Ling, Huasen He, Xiaofeng Jiang
Institution: University of Science and Technology of China
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4413080282