Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs
Year: 2026
Type: conference-paper
Abstract: Although speculative decoding is widely used to accelerate Vision-Language Models (VLMs) inference, it faces severe performance collapse when applied to Video Large Language Models (Vid-LLMs).The draft model typically falls into the trap of attention dilution and negative visual gain due to key-value cache explosion and context window mismatches.We observe a visual semantic internalization phenomenon in Vid-LLMs, indicating that critical visual semantics are implicitly encoded into text hidden states during deep-layer interactions, which renders raw visual inputs structurally redundant during deep inference.To address this, we propose the Sparrow 1 framework, which first utilizes visually-aware text-anchored window attention via hidden state reuse to fully offload visual computation to the target model, and leverages intermediate-layer visual state bridging to train the draft model with semantic-rich intermediate states, thereby filtering out lowlevel visual noise.Additionally, a multi-token prediction strategy is introduced to bridge the training-inference distribution shift.Experiments show that Sparrow achieves an average speedup of 2.82× even with 25k visual tokens, effectively resolving the performance degradation in long sequences and offering a practical solution for real-time long video tasks.* indicates corresponding authors. 1
Institution: National University of Defense Technology
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Awards: G2277931485, G2498301691
OpenAlex ID: W7166815409