Work

SGDRC: Software-Defined Dynamic Resource Control for Concurrent DNN Inference on NVIDIA GPUs

Year: 2025

Type: conference-paper

Abstract: Cloud service providers heavily colocate high-priority, latency sensitive (LS), and low-priority, best-effort (BE) DNN inference services on the same GPU to improve resource utilization in data centers. Among the critical shared GPU resources, there has been very limited analysis on the dynamic allocation of compute units and VRAM bandwidth, mainly for two reasons: (1) The native GPU resource management solutions are either hardware-specific, or unable to dynamically allocate resources to different tenants, or both; (2) NVIDIA doesn't expose interfaces for VRAM bandwidth allocation, and the software stack and VRAM channel architectures are black-box, both of which limit the software-level resource management. These drive prior work to design either conservative sharing policies detrimental to throughput, or static resource partitioning only applicable to a few GPU models.

Source: ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming

DOI: 10.1145/3710848.3710863

Authors: Yongkang Zhang, Haoxuan Yu, Chenxia Han, Cheng Wang, Baotong Lu, Yunzhe Li, Zhifeng Jiang, Yang Li, Xiaowen Chu, Huaicheng Li

Institutions: Hong Kong University of Science and Technology, Chinese University of Hong Kong, Alibaba Group (China), Microsoft Research Asia (China), Shanghai Jiao Tong University, China University of Geosciences, The Hong Kong University of Science and Technology (Guangzhou), Virginia Tech

Language: English

Topic: Cloud Computing and Resource Management

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: green

Funders: University of Alberta, Tsinghua University, Chinese University of Hong Kong

OpenAlex ID: W4408049809

FWCI: 10.5

Cites: 35

Cited by: 9

Related to: 10