Year: 2026
Type: article
Abstract: Deep learning systems often permute tensors distributed across GPUs. PyTorch's all_to_all_nd writes each GPU's result into a separate output buffer, doubling the memory each GPU's share occupies. We design Distributed Cycle Splitter (DCS), an in-place alternative to all_to_all_nd for the same permutations. DCS keeps its total auxiliary memory within 5% of the tensor instead of a full output buffer. It follows the permutation's cycles, writes blocks that must move across GPUs into the other GPU's inbox buffer through symmetric memory, and the receiver unpacks them one group at a time. On two PCIe-connected GPUs with 40 MB to 4 GB tensors, DCS stays within this bound on all 271 shapes and uses 94-98% less auxiliary memory per GPU than all_to_all_nd.
Authors: Jayamanna, Kaveen, Zhang, Xuechen ;
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7237380269
