Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections
Year: 2024
Type: conference-paper
Abstract: Deep learning (DL) jobs use multi-dimensional parallelism, i.e., combining data, model, and pipeline parallelism, to use large GPU clusters efficiently. Long-running jobs may experience changes to their GPU allocation: (i) resource elasticity during training adds or removes GPUs; (ii) hardware maintenance may require redeployment on different GPUs; and (iii) GPU failures force jobs to run with fewer devices. Current DL frameworks tie jobs to a set of GPUs and thus lack support for these scenarios. In particular, they cannot change the multi-dimensional parallelism of an already-running job in an efficient and model-independent way.
Authors: Marcel Wagenländer, Guo Li, Bo Zhao, Luo Mai, Peter Pietzuch
Institutions: Imperial College London, Aalto University, University of Edinburgh
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4404400583