Co-designing Communication Middleware and Deep Learning Frameworks for High-Performance DNN Training on HPC Systems
Year: 2020
Type: article
Abstract: Recent advances in Machine Learning (ML) and Deep Learning (DL) techniques have triggered key success stories in many application domains like Computer Vision, Speech Comprehension and Recognition, and Natural Language Processing.Large-scale Deep Neural Networks (DNNs) are primary drivers of these success stories.However, training complex DNN architectures that consist of millions of trainable parameters is computeintensive.Training is done using a large number of examples (training data set) and can take from weeks to months to achieve state-of-the-art prediction capabilities (accuracy).To achieve higher accuracy, making the DNN deeper and larger has become a common strategy but it also leads to a significantly bigger memory footprint.Thus, DNN training is not only compute-intensive but also a memory-hungry process requiring gigabytes of memory.To accelerate the process of large-scale DNN training, this dissertation is focused on designing high-performance systems that can exploit thousands of CPUs and GPUs for faster training.The novel approach presented in this work is called the co-design of high-performance communication middleware and DL frameworks.Co-design is necessary because of the complexity of the overall execution stack for modern DL frameworks.Broadly, this stack consists of many layers, which start from the application layer followed by the DL framework layer (e.g.TensorFlow).The next layer in the stack is the distributed training I would like to thank Almighty Allah (God) for providing me with the ability, courage, patience, and dedication to complete this work.I would also like to thank many people who have stood by my side for the last six years and made this work possible.This dissertation would not have been possible without the invaluable guidance and support from my advisor, Prof. Dhabaleswar K. (DK) Panda.The toughest challenges encountered throughout this research journey were made easier and manageable not only because of Prof. Panda's guidance but also because I was fully supported through grants and funding that he works really hard for.I have not seen a person who is so hardworking and dedicated in every aspect of work and life yet so calm and peaceful.One can only hope to be as successful as Prof. Panda.Thank you for all your hard work, your patience and resilience, your perseverance and dedication, and your evergreen excitement for research.The love and support I have received from Sidra (my wife and the love of my life) during the Ph.D. program are hard to put in words.This work would have been impossible if she did not provide her unconditional support and love throughout the entire six years of the program.The countless late nights I spent reading and writing papers while she cared for my children; feeding them, changing them, taking them to the playground, putting them to bed, and dropping them to school every day, is something I am deeply beholden and thankful for.Sidra, you are simply the best and I am irrevocably in love with you for who you are and what you do every day that keeps Us going.vi
Author: Awan, Ammar Ahmad
Institution: The Ohio State University
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6980663967
