Applying Certified L∞-Bounded Compression to LLM Serving: Portable Saved State, Shared Pools and Mixed-Generation GPUs
Year: 2026
Type: article
Abstract: Earlier work (Parts~I and~II of the author's series on certified $L_\infty$-bounded compression) compresses a language model's weights and key--value (KV) cache under a certified $L_\infty$ contract: every stored number is within a stated tolerance of the original. This paper asks what the contract makes possible in a \emph{serving} system. A saved prefix (the KV cache and recurrent state of the beginning of a conversation) is then \emph{data defined by its numbers}, not by the GPU that made it. We use this in four ways: a transfer signature and per-piece hashes that decide whether a saved prefix may be used; a copy-or-recompute rule priced from a measured link speed; a host-RAM pool shared by the servers of a machine and a RAM-only machine that holds copies and overflow; and layer splits over cards and machines. On a DGX Spark (Blackwell) and a V100 (Volta), a prefix of 12\,288 tokens (496~MB) saved on one was restored on the other, giving the same greedy answer in 10.3~s against 40.2~s for computing it cold. A 27-billion-parameter model whose pack is 13.6~GB runs on one 16~GB V100 with host-RAM offload. All numbers are single runs on three machines and a Wi-Fi link; the large-server layouts we draw are simulations, not results.
Author: SAM, Chi Ngai
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7234484265
