Work

Benchmarking the performance of GPT-2 type applications on GPU-accelerated computing resources

Year: 2024

Type: article

Abstract: In this report we look at eight GPU-accelerated systems representing across-section of the available Tier 2 HPC systems in the UK. These includea mixture of GPU-accelerated platforms from NVIDIA, AMD and Intel. For each system, we perform a set of benchmarking experiments by training a GPT-2 model using a mixture of parameters and hyperparameters. These include model size, number of GPUs, floating point data type, training datasize, distribution strategies and batch size. Our interest is in performance measured by time taken to complete an epoch of training rather than convergence speed. We also measure memory usage. The overall aim is to compare systems in order to provide researchers intending to perform AI training with some benchmarks for what to expect in terms of training speed for a modelin a non-optimised real-world scenario.

Source: Zenodo (CERN European Organization for Nuclear Research)

DOI: 10.5281/zenodo.13349540

Authors: Lazauskas, Tomas, Llewellyn-Jones, David

Institution: The Alan Turing Institute

Language: English

Topic: Parallel Computing and Optimization Techniques

Subfield: Hardware and Architecture

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: Alan Turing Institute, UK Research and Innovation

Awards: G1309768829, G2025282107, G3560160643, G7804519787