Reservoir of Importance: Learning Semi-Structured Sparsity with Differentiable Subset Sampling
Year: 2026
Type: conference-paper
Abstract: Semi-structured N :M sparsity has emerged as a practical direction for accelerating large language models (LLMs).However, existing learnable-mask approaches incur substantial parameter and memory overhead, limiting their scalability to large models and aggressive sparsity regimes.In this work, we revisit semistructured pruning from a perspective that reconciles efficiency with scalability.We propose Reservoir of Importance (RoI) 1 , a lightweight semi-structured pruning framework that learns sparsity masks through differentiable subset sampling.Unlike prior methods that model full categorical distributions over all feasible N :M patterns, RoI introduces a compact-logit parameterization for sparsity mask learning and performs sampling without replacement to select masks, thereby reducing trainable parameters from combinatorial complexity to O(M ).As a result, RoI requires 1.5-8.75×fewer learnable parameters and significantly lower memory cost, while remaining fully aligned with hardware-friendly sparsity patterns.Extensive evaluations across multiple scales of the Qwen2.5 LLM family (0.5-7B parameters) demonstrate that RoI achieves competitive performance with strong memory efficiency, stability, and scalability to more aggressive N :M sparsity patterns, offering a practical path toward efficient LLM deployment.
Authors: Ha Dinh, Xuan Duy Ta, Khoat Than, Khac-Hoai Nam Bui
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7204158490