Year: 2026
Type: conference-paper
Abstract: Large Language Models (LLMs) have rapidly advanced in recent years, scaling up in both parameter count and context length.However, as context windows extend from thousands to hundreds of thousands of tokens, attention computation becomes the dominant source of memory usage and runtime in decoding stages, severely limiting the efficiency and scalability of longcontext LLMs.Sparse attention has emerged as a promising solution, reducing complexity by computing attention over only a subset of context tokens.However, the sparse attention for Multi-head Latent Attention (MLA) which is a variant of standard MHA is rarely studied.In this paper, we introduce RoPE-based Blockwise Sparse Attention (RoBSA), a method designed specifically for MLA during the decoding stage of model inference.RoBSA leverages the decoupled nature of RoPE within MLA to implement token selection in a blockwise manner.RoBSA is a lightweight, training-free, and layer-aware algorithm that can be integrated in a plug-and-play fashion.Our method significantly reduces end-to-end inference latency in the decoding stage by up to 2.55× with minimal accuracy loss compared to full attention in long-context scenarios for very large models.
Authors: Xinyu Shi, Kairong Luo, Zhen Zheng, Wenguang Chen
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Award: G8141470120
OpenAlex ID: W7166827041