Year: 2025
Type: preprint
Abstract: We present SPDL (Scalable and Performant Data Loading), an open-source, framework-agnostic library designed for efficiently loading array data to GPU. Data loading is often a bottleneck in AI applications, and is challenging to optimize because it requires coordination of network calls, CPU-bound tasks, and GPU device transfer. On top of that, Python's GIL (Global Interpreter Lock) makes it difficult to gain performance improvement from multi-threading. We found that when data preprocessing functions release the GIL entirely, it is possible to execute them concurrently in a thread pool, thereby improving the workflow performance. Our benchmark shows that compared to the PyTorch DataLoader, SPDL can iterate through the ImageNet dataset 74% faster while using 38% less CPU and 50GB less memory. When training ViT-B/16 model, SPDL can send data to the GPU at a speed that does not starve the training. Additionally, when using SPDL on Python 3.13t, without changing any code, the throughput is further by improved by 33%, thanks to the disabled GIL. SPDL can improve the performance of current AI model training, and receives further performance improvements when Free-Threaded Python is adopted in production systems. SPDL is available at https://github.com/facebookresearch/spdl.
Source: arXiv (Cornell University)
Authors: Hira, Moto, Puhrsch, Christian, Andrei, Valentin, Malinovskyy, Roman, Lan, Gael Le, Krishnan, Abhinandan, Cummings, Joseph, Bourgin, Victor, Gerasimova, Olga, Martin, Miguel, Gunasekaran, Gokul, Inoue, Yuta, Turner, Alex J, Krishnamoorthi, Raghuraman
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4417167297
Cites:
Cited by:
Related to: