Year: 2026
Type: conference-paper
Abstract: Accelerating the inference of large language models (LLMs) has been a critical challenge in generative AI.Speculative decoding (SD) substantially improves LLM inference efficiency.However, its utility is limited by a fundamental constraint: the draft and target models must share the same vocabulary, thus limiting the herd of available draft models and often necessitating the training of a new model from scratch.Inspired by Dynamic Time Warping (DTW), a classic algorithm for aligning time series, we propose the algorithm TokenTiming for universal speculative decoding.It operates by re-encoding the draft token sequence to get a new target token sequence, and then uses DTW to build a mapping to transfer the probability distributions for speculative sampling.Benefiting from this, our method accommodates mismatched vocabularies and works with any off-the-shelf models without re-training and modification.We conduct comprehensive experiments on various tasks, demonstrating 1.57× speedup.This work enables a universal approach for draft model selection, making SD a more versatile and practical tool for LLM acceleration.The code is available at the link.
Authors: Sibo Xiao, Fu Jinyuan, Zhongle Xie, Lidan Shou
Institution: Zhejiang University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166854755