LLM Inference Acceleration on Edge Devices via Dual-Model Parallel Speculative Decoding
Year: 2026
Type: other
Abstract: This project builds on the finding that the draft model and target model in the speculative decoding architecture can run in parallel to a certain extent. The two models will be deployed on two separate processing units of edge devices such as mobile phones — for example, the draft model will run on the NPU and the target model on the GPU. Systematic design will be adopted to minimize hardware resource contention between the two models, so as to reduce LLM inference latency.
Source: California Digital Library
DOI: 10.48321/d1b6f93cf1
Author: Zhuohan Huang
Institution: Chinese University of Hong Kong
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7171506451
Cites:
Cited by:
Related to: