Year: 2026
Type: conference-paper
Abstract: The autoregressive inference in large language models requires repeated computation across transformer layers.While caching intermediate key-value (KV) pairs eliminates redundancy, it introduces severe memory overhead, particularly in long-context settings.Most existing cache compression methods operate solely on either quantization or eviction, based on importance estimation of cached data.However, they are limited by coarse compression choices and inaccurate importance assessment, leading to suboptimal inference quality.To address this, we propose HqeKV, a hybrid compression framework built on both quantization and eviction, offering finer-grained compression options that adapt smoothly to the varying importance of cached KV pairs.An integrated optimizer automatically selects the best compression action for each cached element, maximizing quality while insulating end-users from tedious low-level tuning details.We further design a joint K-V importance metric to provide more accurate importance assessment results so that the optimizer can make smarter decisions.Additionally, HqeKV supports flexible conversion policies across multiple quantization precision levels, to further reduce quality degradation.Extensive experiments show that HqeKV improves output quality under the same memory constraints, outperforming stateof-the-art alternatives.Code is available at https://github.com/skywclouds/HqeKV.
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Award: G3791426778
OpenAlex ID: W7166850384