Work

Adaptive hybrid speculative decoding for accelerating large language model inference

Year: 2026

Type: article

Abstract: Processing long sequences efficiently is crucial for modern large language models (LLM), yet serving these models incurs high inference costs due to computational bottlenecks in speculative decoding. Existing speculative decoding methods leverage fixed-granularity draft–verification strategies, which introduce a trade-off between inference speed and accuracy: a larger draft step accelerates decoding but increases verification overhead, while a smaller step ensures correctness but reduces efficiency. To overcome this trade-off, we propose Adaptive Hybrid Speculative Decoding (AHSD) , a novel framework built upon two key algorithmic innovations: Adaptive Feedback Thresholds and Confidence-Driven Two-Stage Token Processing, implemented efficiently via a bucketed batching optimization. First, the adaptive feedback threshold dynamically adjusts rollback and validation by monitoring the draft model’s confidence in real time, if confidence exceeds the threshold, verification is skipped; otherwise, a refining check is triggered, maximizing first-pass success while minimizing false positives. Second, the confidence-driven two-stage token processing decouples draft generation from verification, allowing high-confidence tokens to bypass verification entirely. To ensure these algorithmic innovations execute efficiently on modern GPUs, we employ a bucketed batching strategy that groups sequences by dynamically chosen block sizes , pads each homogeneous batch to its maximum length with a unified attention mask, and executes all sequences in parallel, thereby eliminating scheduling overhead and branch divergence caused by length variability.This sharply reduces re-model frequency and enhances parallelism. Comprehensive simulations show that AHSD achieves up to speedup and throughput gain over existing schemes, making it suitable for applications that can tolerate minor quality loss for substantially faster inference.

Source: Neurocomputing

DOI: 10.1016/j.neucom.2026.133760

Authors: Yang Yong, Ting Ting Yang, Shao Shuai Gao, Ye Hua Yin

Institutions: Chinese Academy of Sciences, Peng Cheng Laboratory, University of Chinese Academy of Sciences

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Estimated APC paid: $2,670

Funder: Peng Cheng Laboratory