HCSpec: Two-Tier Horizontal Cascade Speculative Decoding for High-Efficiency Large Language Model Inference
Year: 2026
Type: conference-paper
Abstract: Speculative decoding accelerates large language model (LLM) inference by using a draft model to propose token candidates for parallel verification by the target model.However, current state-of-the-art self-distilled draft models adopt a homogeneous architecture across all drafting positions, failing to account for a critical empirical observation: the expected utility of drafting decays rapidly after the initial positions.To exploit this imbalance, we propose Two-tier Horizontal Cascade Speculative Decoding (HCSpec), a novel framework that organizes heterogeneous, positionspecialized draft modules into a horizontal cascade.The first tier employs a dual-layer, dual-path transformer that enhances early-step fidelity by decoupling token-logit prediction from recurrent feature propagation, while the second tier adopts a lightweight single-layer transformer that deliberately trades marginal accuracy for improved efficiency at later drafting steps.Extensive experiments on Qwen series models and Llama3.1-8B-Instruct,across multiple tasks and diverse inference configurations, demonstrate that HCSpec consistently outperforms the previous state-of-the-art (EAGLE-3).It delivers 15-30% higher end-to-end speedup over EAGLE-3 and achieves up to 3.72x acceleration over vanilla autoregressive decoding.Our code is provided in the supplementary materials.
Authors: Yizhou Zhang, Siming Chen, Hao Ye, Erhu Feng
Institutions: Shanghai Jiao Tong University, Lanzhou University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166886618