Year: 2025
Type: conference-paper
Abstract: Speculative decoding has proven to be a powerful method for speeding up autoregressive inference by allowing tokens to be generated in parallel using a draft-then-verify approach. However, the current techniques tend to rely on fixed lengths for drafting and strict criteria for verification, which limits their flexibility when faced with different levels of model uncertainty or varying input complexities. In this paper, we propose a new framework for speculative decoding, built on the idea of adjusting drafting based on confidence levels. By using entropy and margin-based uncertainty measures derived from the drafter’s output distribution, the approach can dynamically change the number of tokens it generates at each step. This adaptability not only reduces the need for rollbacks but also makes better use of available resources, all while ensuring that the output remains accurate. The process for verifying these drafts is also adjusted based on confidence signals, which makes it easier to accept tokens without sacrificing the quality of the generation. When tested on machine translation and summarization tasks, this method delivers impressive speedups over standard speculative decoding and even improves BLEU and ROUGE scores. This approach provides a flexible, easy-toimplement solution for more efficient and reliable decoding in large language models, especially when faced with uncertain conditions.
Authors: Jaydip Sen, Subhasis Dasgupta, Hetvi Waghela
Institution: Praxis Business School
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4416924739
