How far from optimal is recursive rejection sampling? A tight bound for multi-draft speculative decoding with independent drafts
Year: 2026
Type: preprint
Abstract: Editorial revision; no result changed. Corrected credit: the optimal accepted-token law a* for iid drafts (the cumulative-minimum construction used by KEY) is due to Thomas and Pal, "Global Resolution" (arXiv:2511.15898; their Theorems 6.1-6.2, Lemma 6.3 and eq. (7)). Their method is now described accurately: it is for iid drafts and is accurate to a set tolerance, with a fallback. The stale "Partial Resolution" workshop citation is removed; it is the same work. What is new in KEY is the closed-form selection law r*, its exact realisation by interval keys, and the lazy variant. Two small proof-step slips in the tree section were corrected (the telescoping derivation of RRS's per-token acceptance, and the strictness step in the proof of the reach-probability theorem); no theorem changed. Speculative decoding speeds up a large language model: a small draft model proposes tokens, and the large model verifies them without changing its output distribution. With k independent drafts per position, SpecInfer's recursive rejection sampling (RRS) tests drafts one at a time, with no guarantee we know of beyond 1/k. We prove that RRS achieves at least a fraction rho_k = 1 - max_{0<=a<=1}(a^k - a^{2k-1}) of the optimal acceptance, and that this is tight: rho_2 = 23/27, and rho_k decreases to 3/4. The proof writes RRS as one scalar recursion and lumps the tokens into two groups by the sign of the final residual, which can only lower RRS's acceptance and only raise the optimum. We also give KEY, an exact optimal verifier: after one sort and an interval-placement step, each draft draws a random key, and the draft with the smallest key is verified. On two small model pairs (temperature 1, four drafts), the mean acceptance gap of RRS is only 0.033, yet RRS rejects 1.35 times as often as the optimal verifier (pooled). The supplement contains all scripts and results needed to reproduce every computed number (MIT license). Version 1.5: editorial revision of version 1.4 with the credit correction above. Changes: a single title without subtitle; a shorter abstract and introduction; a four-item results box. The K-SEQ and RSD details, most experiment results and the KEY formulas moved to the appendices (main text 23 pages, down from 30), and the practical guidance is merged into one section. Version history was removed from the paper. Theorem numbers changed. The supplement is the version 1.4 supplement with its README renumbered and internal process wording removed from three note headers. Version 1.4: tight rejection curve of RRS; KEY, an exact optimal verifier for iid drafts; experiments on where the gap sits and what it costs (pre-registered statements S1-S6: S1 and S5 false, S2-S4 and S6 true; exploratory E1b and E3c), including tokens per target call in idealised trees. Version 1.3: editorial restructuring of version 1.2 (no result changed). Version 1.2: token trees, SpecHub at k=2, progress on the 3/4 conjecture for drafting without replacement. Version 1.1: RSD never worse than RRS; K-SEQ tight constant for every k; RDR equals RRS on two tokens. Version 1.0: the tight ratio rho_k for RRS.
Authors: Baliga, Kundan, Xie, Aiden, Byju, Aditya
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225430377