Year: 2026
Type: conference-paper
Abstract: Query-conditioned sparse KV selection reduces long-context inference cost by attending to a subset of historical KV entries at each decoding step.Existing selectors must wait for the target-layer query before scoring candidates and generating indices, placing routing on the immediate pre-attention path.We introduce EARLYSELECTKV, a trainingfree transformation that uses proxy queries from earlier hidden states to precompute selected KV indices while retaining the real target-layer query for final attention.Tests with RocketKV, Quest, and Loki show no consistent pattern of quality degradation across the evaluated settings.In the primary Rock-etKV evaluation, ES-RocketKV differs by +0.018 points on LongBench, approximately -0.01 points on RULER, and +0.25 percentage points on representative NIAH/Passkey settings.In a TensorRT-LLM deployment with CPU-offloaded routing, ES-RocketKV improves end-to-end output throughput by up to 12.4% and reduces GPU-resident memory by 11.1% of the combined KV/K-summary pool.This makes routing timing an explicit optimization target alongside KV subset selection in sparse long-context decoding.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
Award: G2729455034
OpenAlex ID: W7224599385