Year: 2026
Type: conference-paper
Abstract: To reduce memory consumption during LLM inference, prior works have proposed numerous methods that focus on KV cache pruning based on various designed criteria.While these techniques often accomplish lossless memory reduction on many datasets, they often hinge on an under-emphasized condition: an input/domain-specific budget threshold needs to be pre-determined to achieve the optimal performance.However, such input-sensitive design may be considerably limited in real-world scenarios, as open-domain inputs span diverse domains, lengths and difficulty levels, without clear boundaries for threshold selection.As a result, the dependence of such input-sensitive threshold can be a fundamental limitation that causes large degradation on arbitrary inputs.In this work, we propose a new objective that lifts the threshold constraints for robust KV compression, advocating for "threshold-free" methods that adaptively adjust budget allocation while preserving full-cache performance.We then propose a novel method, ReFreeKV, serving as the first instantiation of this objective.Extensive experiments across 13 datasets with diverse context lengths, task types, and model sizes demonstrate its efficacy and efficiency.
Institution: Nanjing University of Aeronautics and Astronautics
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166810043