Work

FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference

Year: 2025

Type: conference-paper

Abstract: Large Language Models (LLMs) face challenges for on-device inference due to high memory demands. Traditional methods to reduce memory usage often compromise performance and lack adaptability. We propose FlexInfer, an optimized offloading framework for on-device inference, addressing these issues with techniques like asynchronous prefetching, balanced memory locking, and flexible tensor preservation. These strategies enhance memory efficiency and mitigate I/O bottlenecks, ensuring high performance within user-specified resource constraints. Experiments demonstrate that FlexInfer significantly improves throughput under limited resources, achieving up to 12.5 times better performance than existing methods and facilitating the deployment of large models on resource-constrained devices.

DOI: 10.1145/3721146.3721961

Authors: Hongchao Du, Shangyu Wu, Arina Kharlamova, Nan Guan, Chun Jason Xue

Institutions: City University of Hong Kong, Mohamed bin Zayed University of Artificial Intelligence

Language: English

Topic: Parallel Computing and Optimization Techniques

Subfield: Hardware and Architecture

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

OpenAlex ID: W4409047785

FWCI: 5.443

Cites: 16

Cited by: 6

Related to: 10