Year: 2019
Type: article
Abstract: Training deep neural networks is a computationally intensive task. The training time can be shortened by using hardware accelerators such as Graphics Processing Units (GPUs). Therefore, multi-GPU systems are used to speed up the training process. During the training process data needs to be copied from the CPU to the GPU and also between the GPUs. This thesis evaluates the performance of different on-node GPU interconnects: PCIe and NVLink. Microbenchmarks are designed and conducted to measure the performance of these interconnects for basic operations involved in training deep neural networks. The practical impact of using different interconnects when training various deep neural networks on multi-GPU systems is measured for using PCIe interconnects and for using NVLink interconnects. These experiments are conducted on different multi-GPU systems that have distinct interconnect systems. A classification approach is defined, which is used to classify the training workloads of deep neural networks into rather communication- or computation-intensive workloads. This classification is necessary to understand how much a certain training workload would benefit from a high-performance interconnect system.
Author: Nane-Maiken Zarges
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W2950056902