Work

Recovering Cache Reuse in Hybrid LLMs Through Checkpoint Alignment

Year: 2026
Type: preprint
Abstract: Models that combine attention and recurrent layers can reduce inference memory requirements. This hybrid cache approach needs saved checkpoints to reuse earlier computation. In contrast, standard attention caches retain keys and values for individual tokens. We identified a mismatch between how coding agent harnesses replace task state and where hybrid caches save checkpoints, causing unchanged conversation to be processed again. We examined 17 harnesses and measured this behavior through deployed providers, vLLM and llama.cpp. Our approach recovers cache reuse while preserving compact replacement of task state. A controlled comparison on nine traces shows the tradeoff: retaining or repeatedly fetching state enlarged prompts, while an earlier anchoring method reduced hybrid recomputation by 70% versus unprotected replacement. Client side padding reduced median recomputation by 52% across eight deployed sessions. A server side checkpoint hint leaves prompt tokens unchanged and adds no model request. On the hybrid model Qwen3.5-4B, it reduced recomputation by 20% and time to first token by 16% in serial replay. With four concurrent sessions, it reduced mean block 95th percentile time to first token by 41%. Full response time did not detectably improve in serial replay. Output differences do not establish a change in task quality. These results show how checkpoint placement can recover reuse without accumulating old task snapshots. Author preprint. This manuscript is under peer review at MLSys 2027 and has not been accepted for publication.
Language: English
Open Access status: green
OpenAlex ID: W7233555174
Cites:
Cited by:
Related to: