Year: 2025
Type: conference-paper
Abstract: Quantization has played a significant role in enabling large language models to operate efficiently. Quantization-Aware Training (QAT) compensates for the loss incurred during the quantization process through training, and has demonstrated promising results. However, in the case of extremely low-bit quantization, such as 3 bits, the performance of QAT degrades significantly. In this paper, we delve into the challenges associated with data selection and training approach. Specifically, we have initially analyzed which type of data yields better results when applying ultra low-bit QAT to base or chat models. Building on this analysis, we further propose an iterative training approach that enhances the stability of model quantization at extremely low-bit configuration. Experimental evaluations demonstrate the effectiveness of the proposed method.
Authors: Jiangcun Du, Renren Jin, Wuwei Huang, Wei Liu, Jian Luan, Deyi Xiong
Institutions: Tianjin University, Xiaomi (China)
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Award: G501308975
OpenAlex ID: W4410636396