Work

Medusa: Accelerating Serverless LLM Inference with Materialization

Year: 2025

Type: conference-paper

Abstract: Serverless is a promising paradigm to provide scalable, cost-efficient, and easy-to-use model inference services. However, the cold start of model inference functions requires loading models to the devices, which incurs high latencies and undermines the benefits of serverless computing. In LLMs, things get even worse since two extra stages are introduced: a KV cache initialization stage that profiles and anticipates memory reservation for KV cache, and a capturing stage which dynamically constructs CUDA graphs for different batch sizes. Both stages are paramount to the inference performance, but become the main culprit of cold start latency.

Source: ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)

DOI: 10.1145/3669940.3707285

Authors: Shaoxun Zeng, Minhui Xie, Shiwei Gao, Youmin Chen, Youyou Lu

Institution: Tsinghua University

Language: English

Topic: Cloud Computing and Resource Management

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: Universitas Brawijaya

Award: G2948749109