Year: 2026
Type: conference-paper
Abstract: Speculative decoding has rapidly emerged as a leading approach for accelerating language model (LM) inference, as it offers substantial speedups while yielding identical outputs.This relies upon a small draft model, tasked with predicting the outputs of the target model.Stateof-the-art speculative decoding methods use a draft model comprising a single decoder layer and output embedding matrix, with the latter dominating drafting time for the latest LMs.Recent work has sought to address this output distribution bottleneck by reducing the vocabulary of the draft model.While this can improve throughput, it compromises speculation effectiveness when the target token is out-ofvocabulary.In this paper, we argue for vocabulary speculation as an alternative to a reduced vocabulary.We propose SpecVocab, an efficient and effective method that selects a vocabulary subset per decoding step.Across a variety of tasks, we show that SpecVocab can achieve a higher acceptance length than state-of-theart speculative decoding method, EAGLE-3.Notably, this yields up to an 8.1% increase in average throughput over EAGLE-3. 1
Institution: University of Sheffield
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Funder: UK Research and Innovation
Awards: G2831959038, G5306878388
OpenAlex ID: W7166825217