Work

GM-Sparse-LM: Activation-Sparse FFN Execution for Memory-Bandwidth-Bound CPU Inference

Year: 2026
Type: preprint
Abstract: Decoding a dense language model on a commodity PC with single-channel memory isbounded by RAM bandwidth, not compute. On an 8 GB single-channel DDR4-3200 laptop(about 11 GB/s effective decode bandwidth), a dense 4B Q4 model is capped at roughly4 tokens per second. This work measures the constraint on real hardware, studies FFNactivation sparsity on Qwen3-0.6B across all 28 layers, and proposes GM-Sparse-LM:an importance-sorted dual-precision FFN layout with masked cold-block execution anda tiered training pipeline (SiLU-to-ReLU conversion, ProSparse-style regularizedfine-tuning, calibration) projected to reach roughly 2.4x speedup at 90 percentsparsity. Measured contributions include a validated byte-level conversion path forGGUF, converted-model benchmarks (the expert-form model reads 25 percent of FFNbytes and gains 1.21x in the bandwidth-bound regime), a measured training-freequality cliff at roughly 37 percent sparsity that matches the sparsity study'soperating point, an implementation finding that explains why prior activation-sparsity systems all ship custom kernels, and eight quantified negative results.Trained-model quality evaluation is in progress.
Author: Arun, Sahil
Language: English
Open Access status: green
OpenAlex ID: W7237355012
Cites:
Cited by:
Related to: