Work

Performance Modeling and Device Placement Technique for Parallel Training Deep Neural Networks in Heterogeneous GPU Clusters

Year: 2019
Type: dissertation
Abstract: We introduce a performance model and a device placement algorithm for pipeline parallelism. Our performance model basically removes off-line profiling procedure which is used to estimate training speed of a Deep Neural Network(DNN) model training. Our model can be applied to data parallelism, model parallelism, and pipeline parallelism as well. Also, our device placement technique can exploit our performance model to avoid off-line profiling. Our prediction model only takes less than a few seconds while our device placement algorithm using the results from modeling shows better training speed than data parallelism as a baseline. Our heterogeneous-aware device placement achieves 1.12x for VGG model compared to the baseline partition scheme which does not consider heterogeneous GPU performance. Our performance model predicts one iteration time of large-scale DNN models such as Alexnet, Inception4, VGG16, Overfeat with an average error rate of 30.12%.
Author: Kim, DoHyung
Language: English
Open Access status: green
OpenAlex ID: W7004955369
Cites: 14
Cited by:
Related to: 10