Work

ChiselMark: Efficient and Detectable Watermarking for Large Language Models under Speculative Sampling

Year: 2026
Type: preprint
Abstract: Generation-time watermarking has become a central safeguard for tracing the outputs of hosted large language models (LLMs) and protecting providers' intellectual property against unauthorized use, yet it must coexist with the increasingly optimized systems through which these models are deployed in production. One such system is speculative sampling, a widely used paradigm for accelerating LLM generation that is now embedded in most production LLM-serving stacks. Here, we study KGW-style logit-reweighting watermarks under speculative sampling and show that straightforward integration systematically erodes the acceleration that speculative sampling is meant to deliver, even when the intended watermarked output distribution is preserved at the token level. We trace this erosion to a latent assumption that current KGW-style methods inherit from standard autoregressive decoding, namely that the distribution being reweighted is also the one whose statistics determine the cost of a write. Under speculative sampling this assumption breaks, because the cost of a write is paid by the proposal side, by the draft distribution at the token level and by the already-constructed candidate tree at the feature level, rather than by the reweighted target. Guided by this understanding, we introduce ChiselMark, a sparse, cost-aware, and architecture-aware watermarking framework that decomposes watermark strength into three distinct controls: where the watermark may be written, whether a particular write is predicted to be locally affordable, and at which stage of the decoding architecture the write enters, while remaining detectable from the generated text, a secret key, and public detector parameters. Across token-level speculative sampling, Medusa, and three generations of EAGLE on multiple model scales and tasks, ChiselMark recovers much of the speculative efficiency lost to dense watermarking while keeping detectability competitive with existing watermarks, and retains detectability under text edits. These findings indicate that practical mechanisms for LLM traceability and controlled deployment should be co-designed with the inference architectures on which they operate, rather than treated as orthogonal interventions at the next-token distribution.
Language: English
Open Access status: green
OpenAlex ID: W7235604138
Cites: 38
Cited by:
Related to: 10