Work

10C ache : Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Training

Year: 2025

Type: conference-paper

Abstract: Training large language models (LLMs) in the cloud faces growing memory bottlenecks due to the limited capacity and high cost of GPUs. While GPU memory offloading to CPU and NVMe has made large-scale training more feasible, existing approaches suffer from high tensor migration latency and suboptimal device memory utilization, ultimately increasing training time and cloud costs. To address these challenges, we present 10Cache, a resource-aware tensor caching and migration system that accelerates LLM training by intelligently coordinating memory usage across GPU, CPU, and NVMe tiers. 10Cache profiles tensor execution order to construct prefetch policies, allocates memory buffers in pinned memory based on tensor size distributions, and reuses memory buffers to minimize allocation overhead.

Source: ACM Symposium on Cloud Computing (SoCC)

DOI: 10.1145/3772052.3772236

Authors: Sabiha Afroz, Redwan Ibne Seraj Khan, Hadeel Albahar, Jingoo Han, Ali R. Butt

Institutions: Virginia Tech, Kuwait University

Language: English

Topic: Parallel Computing and Optimization Techniques

Subfield: Hardware and Architecture

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: National Science Foundation

Award: Collaborative Research: CNS Core: Medium:HardLambda: A new FaaS Abstraction for Cross-Stack Resource Management in Disaggregated Datacenters