Work

ALISE: Accelerating Large Language Model Serving with Speculative Scheduling

Year: 2024

Type: conference-paper

Abstract: Large Language Models (LLMs) represent a revolutionary advancement in the contemporary landscape of artificial general intelligence (AGI). As exemplified by ChatGPT, LLM-based applications necessitate minimal response latency and maximal throughput for inference serving. However, due to the unpredictability of LLM execution, the first-come-first-serve (FCFS) scheduling policy employed by current LLM serving systems suffers from head-of-line (HoL) blocking issues and long job response times.

DOI: 10.1145/3676536.3676659

Authors: Youpeng Zhao, Jun Wang

Institution: University of Central Florida

Language: English

Topic: Cloud Computing and Resource Management

Subfield: Information Systems

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Funder: National Science Foundation

Awards: SHF: Small: Taming Huge Page Problems for Memory Bulk Operations Using a Hardware/Software Co-Design Approach, SHF: Small: Revamping I/O Architectures Using Machine Learning Techniques on Big Compute Machines

OpenAlex ID: W4409282468

FWCI: 0.769

Cites: 6

Cited by: 2

Related to: 10