Work

Cacheback: Speculative Decoding With Nothing But Cache

Year: 2025

Type: conference-paper

Abstract: We present Cacheback Decoding, a trainingfree and model-agnostic speculative decoding method that exploits the locality in language to accelerate Large Language Model (LLM) inference.Cacheback leverages only Least Recently Used (LRU) cache tables of token n-grams to generate draft sequences.Cacheback achieves state-of-the-art performance among comparable methods despite its minimalist design, and its simplicity allows easy integration into existing systems.Cacheback also shows potential for fast adaptation to new domains.

Source: Conference on Empirical Methods in Natural Language Processing (EMNLP)

DOI: 10.18653/v1/2025.emnlp-main.1581

Authors: Zhiyao Ma, In Gim, Lin Zhong

Institution: Yale University

Topic: Natural Language Processing Techniques

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Funder: National Science Foundation

Award: AI Institute for Edge Computing Leveraging Next Generation Networks (Athena)

OpenAlex ID: W4416035406