Mitigating the impact of activation smoothing on weights: A channel permutation and smoothing-based LLMs quantization method
Year: 2026
Type: article
Abstract: Quantization has emerged as an effective technique to reduce the memory requirement of Large Language Models (LLMs). However, we observed that activation smoothing destroys the original flat distribution of weights and existing methods struggle to smooth activation outliers with extremely large magnitudes, resulting in increased quantization errors. To overcome these challenges, we propose CHAMP-Q , a novel quantization method that relies on two strategies: (1) To mitigate the impact of activation smoothing on weights, a multi-dimensional feature-aware channel permutation (MFCP) strategy is designed to alleviate intra-group weight variances by permuting similar channels to adjacent positions in the weight matrix, thereby reducing the group-wise weight quantization error. (2) To reduce the quantization errors caused by activation outliers, a hybrid numerical smoothing (HNS) strategy is proposed to suppress activation outliers by selectively applying different smoothing strategies based on their magnitudes. Furthermore, we implement a W4A8 quantization framework. The experimental results demonstrate that CHAMP-Q enables 4-bit weight and 8-bit activation quantization with less accuracy degradation compared to existing outlier smoothing methods, and achieves up to 1.74 \(\times \) memory savings and 1.45 \(\times \) inference speedup compared to the original model.
Authors: Yanyao Guan, Yunhe Cui, Langtao Zhao, Guowei Shen, Yi Chen, Chun Guo
Institution: Guizhou University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,990
Award: G7321023181
OpenAlex ID: W7133720195