Work

Benchmarking, Profiling and White-box Performance Modeling for DNN Training

Year: 2022
Type: dissertation
Abstract: Training a modern deep learning model is extremely time-consuming. The software/hardware deployments that machine learning (ML) programmers use in practice are widely diverse.Differences in algorithms, hardware features, or even software versions could all shift the performance bottlenecks of practical deployments. Therefore, it usually requires extensive amount of domain knowledge and engineering effort for ML programmers to profile and pinpoint ad-hoc performance bottlenecks. In this thesis, we explore the approach of performance modeling in the context of ML workloads to enable efficient optimizations.Our key idea is to decompose a training workload into atomic traces of heterogeneous hardware components, and properly structure these traces to build a complete profile. Using this profile, we allow ML practitioners to effectively identify the performance bottlenecks and efficient optimizations for any given software/hardware deployments. In summary, this thesis makes the following major contributions. First, we propose a new benchmark suite for DNN training workloads, called TBD, that contains nine mainstream DNN models, covering six major DNN applications. We also propose a set of performance metrics that can directly indicate performance bottlenecks on CPU/GPU, and build a tool chain that enables end-to-end profiling for DNN training workloads. Second, we propose a new performance predictor (Daydream), that allows users to accurately and effectively estimate the efficacy of various optimization techniques for DNN training workloads. Daydream utilizes a dependency graph approach to model the hardware execution, which tracks the task dependencies at kernel level and addresses several unique challenges in the ML context. Third, we propose two tensor compilers, Sokoban and Roller, that can generate high-performance tensor programs in seconds.They view the underlying architecture as a data processing pipeline, and the computation of an operator as independent tiles. Sokoban is a search-based compiler that utilizes an end-to-end cost model based on the pipeline model, while Roller is a construction-based compiler that exploits heuristics from the pipeline model and directly constructs efficient schedules. Both compilers manage to generate tensor programs that are comparable against manually-written libraries and state-of-the-art compilers, as well as reduce the compilation time to seconds.
Author: Zhu, Hongyu
Language: English
Open Access status: green
OpenAlex ID: W7132970710
Cites: 190
Cited by:
Related to: 9