Reconfigurability within Collective Communication Algorithms
Year: 2025
Type: conference-paper
Abstract: With the explosive growth of AI applications, models and datasets continue to evolve rapidly. Today's Deep neural networks (DNN) consist of billions of parameters, requiring them to utilize many high-performance neural processing units (NPUs, such as GPUs or TPUs). Many large cloud providers use Torus topology as the interconnect for these large-scale training workloads (Google TPU, AWS Trainium, etc.). Resource allocations to multiple tenants in a Torus network can result in topology slices of different sizes, and model training on these torus slices will often result in 1D, 2D, or 3D logical communication. In this paper, we first analyze the performance of state-of-the-art collective communication algorithms on various sizes of static torus topology slices. Then, we quantify the benefits that dynamically reconfigurable torus slices can bring to the state-of-the-art collective communication algorithms.
Authors: Rukshani Athapathu, George Porter
Institution: University of California San Diego
Language: English
Subfield: Computer Networks and Communications
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4413922056