Work

Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation

Year: 2023

Type: conference-paper

Abstract: We propose Speculative Decoding (SpecDec), for the first time ever 1 , to formally study exploiting the idea of speculative execution to accelerate autoregressive (AR) decoding.Speculative Decoding has two innovations: Spec-Drafter -an independent model specially optimized for efficient and accurate drafting -and Spec-Verification -a reliable method for verifying the drafted tokens efficiently in the decoding paradigm.Experimental results on various seq2seq tasks including machine translation and abstractive summarization show our approach can achieve around 5× speedup for the popular Transformer architectures with comparable generation quality to beam search decoding, refreshing the impression that the draft-then-verify paradigm introduces only 1.4×∼2× speedup.In addition to the remarkable speedup, we also demonstrate 3 additional advantages of SpecDec, revealing its practical value for accelerating generative models in real-world applications.Our models and

DOI: 10.18653/v1/2023.findings-emnlp.257

Authors: Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, Zhifang Sui

Institutions: Peking University, Microsoft Research Asia (China)

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Funders: Microsoft Research, National Natural Science Foundation of China, Microsoft Research Asia, National Key Research and Development Program of China

Awards: G3077791970, G7138386155

OpenAlex ID: W4389519819