Work

Accelerating LLM Serving for Multi-turn Dialogues with Efficient Resource Management

Year: 2025

Type: conference-paper

Abstract: Although there have been significant efforts to make LLM serving efficient, we observe two limitations of current state-of-the-art serving frameworks in handling multi-turn dialogues between users and assistants, particularly in chat scenarios. First, existing LLM frameworks incur substantial computational overhead in recomputing attention keys and values (KVs) for understanding context across multiple turns of user queries. Second, as the prompt length of user queries is amplified due to multi-turns, a first-come-first-served (FCFS) scheduling policy often causes head-of-line blocking issues, leading to underutilization of GPU resources.

Source: ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)

DOI: 10.1145/3676641.3716245

Authors: Jinwoo Jeong, Jeongseob Ahn

Institution: Korea University

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: Universitas Brawijaya

Award: G1704988562