EC-SpMV: Toward Efficient SpMV in Sparse LLMs via Block Extraction and Compressed Storage
Year: 2026
Type: article
Abstract: Sparse Matrix-Vector Multiplication (SpMV) has become a critical performance bottleneck in the local deployment of sparse Large Language Models (LLMs), where inference predominantly operates on workloads during the decoder phase with a batch size of one. Existing SpMV kernels and sparse matrix formats, originally designed for scientific computing, fail to exploit the unique structure patterns inherent in sparse LLMs, resulting in suboptimal performance and excessive storage overhead. This article presents EC-SpMV, a GPU-optimized SpMV approach for accelerating sparse LLM inference. EC-SpMV introduces (1) a hierarchical block extraction algorithm that captures multiple granularities of block structures within sparse LLMs, and (2) a novel compressed sparse format (EC-CSR) that employs delta indexing to reduce storage overhead and enhance memory access efficiency. Evaluated on sparse weight matrices from LLaMA and OPT models across sparsity levels from 30% to 90%, EC-SpMV achieves up to 7.47× speedup over state-of-the-art SpMV libraries and reduces storage overhead by up to 58.5% compared with CSR.
DOI: 10.1145/3821427
Authors: Junqing Lin, Jingwei Sun, Mingge Lu, Guangzhong Sun
Institution: University of Science and Technology of China
Language: English
Subfield: Hardware and Architecture
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W7169536690