Work

RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding

Year: 2026

Type: conference-paper

Abstract: Autoregressive decoding in Large Language Models (LLMs) generates one token per step, causing high inference latency.Speculative decoding (SD) mitigates this through a guessand-verify strategy, but existing training-free variants face trade-offs: retrieval-based drafts break when no exact match exists, while logits-based drafts lack structural guidance.We propose RACER (Retrieval-Augmented Contextual Rapid Speculative Decoding), a lightweight and training-free method that integrates retrieved exact patterns with logitdriven future cues.This unification supplies both reliable anchors and flexible extrapolation, yielding richer speculative drafts.Experiments on Spec-Bench, HumanEval, and MGSM-ZH demonstrate that RACER consistently accelerates inference, achieving more than 2× speedup over autoregressive decoding, and outperforms prior training-free methods, offering a scalable, plug-and-play solution for efficient LLM decoding.Our source code is available at https://github.com/hkr04/RACER.

Source: Findings of the Association for Computational Linguistics: ACL (Findings of ACL)

DOI: 10.18653/v1/2026.findings-acl.998

Authors: Zihong Zhang, Zuchao Li, Lefei Zhang, Ping Wang, Hai Zhao

Institutions: Wuhan University, Shanghai Jiao Tong University

Language: English

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid