Work

Jenga: Effective Memory Management for Serving LLM with Heterogeneity

Year: 2025

Type: conference-paper

Abstract: Large language models are widely used but expensive to run. To reduce costs, it is crucial to maximize request batch size through efficient GPU memory management. Existing approaches, such as PagedAttention, struggle with modern LLMs because of the growing heterogeneity in the sizes of models' internal embeddings and attention mechanisms.

Source: ACM SIGOPS Symposium on Operating Systems Principles (SOSP)

DOI: 10.1145/3731569.3764823

Authors: Chen Zhang, Kuntai Du, Shu Liu, Woosuk Kwon, Xiangxi Mo, Yufeng Wang, Xiaoxuan Liu, Kaichao You, Zhuohan Li, Mingsheng Long, Jidong Zhai, Joseph Gonzalez, Ion Stoica

Institutions: University of California, Berkeley, Tsinghua University, University of Chicago

Language: English

Topic: Parallel Computing and Optimization Techniques

Subfield: Hardware and Architecture

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

OpenAlex ID: W7084104154

FWCI: 3.623

Cites:

Cited by: 4

Related to: