Year: 2024
Type: conference-paper
Abstract: The efficiency of Deep Learning (DL) training jobs is critically dependent on dataloaders, which facilitate the transfer of data from storage to DL-accelerated hardware during training. Recent advancements in data loading technology have demonstrated significant improvements, not only in reducing training times but also in introducing capabilities such as seamless integration with cloud storage. This paper examines the dataloader as a distinct component within the DL workflow, offering a detailed analysis of its structure and functionalities. We present a systematic evaluation of various dataloading libraries, investigating their performance across different configurations, including worker count, batch size, GPU scaling, data access patterns and remote loading. The evaluation highlights trade-offs in functionality, usability, and performance. Additionally, we examine the impact of dataset characteristics on data loading performance, showing that throughput decreases exponentially with image resolution. To support ongoing research and practical advancements, we introduce the first open-source benchmarking suite for DL data loading, which allows the community to replicate, extend, and build upon our experiments.
Institution: Yale University
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4406458749