Reload or Recompute? A Calibrated Rule for KV-Cache Reuse Across Commodity GPUs
Year: 2026
Type: preprint
Abstract: When a conversation returns, a server must choose whether reloading its key-value (KV) cache or recomputingits prefix gives a faster first token. We evaluate a break-even bandwidth rule for time to first token (TTFT) thatincludes resident-cache continuation, fixed overhead, and CPU and transfer costs. Building on existing calibrationand compute/load analyses, we timestamp and freeze per-cell predictions after short calibration and evaluateseparate sweeps on commodity and older GPUs, including short contexts and separate CPU-memory and filepaths. Frozen predictions correctly classify 574/636 resolved decisions on L4, 192/206 on A10 x16, and 398/426 onA10G x8. For Qwen2.5-1.5B with a 1,024-token prefix and a 32-token new turn, break-even simulated-bandwidthbrackets rise from T4’s 100–400 MB/s to 800–1,600 MB/s on L4 and A10 x16. Long-context extrapolation canproduce large TTFT errors. We release measurements, frozen predictions, reference code, and a source-linkedclaim ledger.
Author: M, Mohith
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225455683