Work

Dynamic Memory Disaggregation for High-Performance Computing Using RDMA-Aware Schedulers

Year: 2025
Type: article
Abstract: High-performance computing workloads increasingly exceed local memory capacity, causing significant performance degradation. Memory disaggregation offers a promising alternative by allowing compute nodes to access remote memory through high-speed interconnects. This paper proposes DM-Flow, a dynamic memory disaggregation framework with a scheduler that is explicitly aware of RDMA latency, bandwidth, and contention. DM-Flow predicts remote access cost using a lightweight network profiler and dynamically assigns memory pages to local or remote pools based on workload access patterns. A tiered caching mechanism improves locality, while a conflict-avoidance algorithm reduces RDMA queue saturation during peak loads. Evaluation on a 64-node cluster using memory-intensive HPC workloads shows speedups of 15–32% over static disaggregation approaches, with significant reductions in queueing delay. Additionally, the scheduler prevents thrashing during multi-tenant execution. The paper outlines hardware constraints, discusses the trade-off between memory elasticity and predictability, and proposes guidelines for integrating DM-Flow into existing HPC ecosystems.
Author: Elias D. Harrington
Language: English
Open Access status: diamond
OpenAlex ID: W7115071941
FWCI:
Cites: 7
Cited by:
Related to: 10