Year: 2024
Type: conference-paper
Abstract: Recently, quantization of large language models (LLMs) has emerged as a crucial technique for model compression, enabling deployment on edge computing devices and enhancing their efficiency. Techniques such as LLM.int8 are widely utilized in the fields of model compression and fine-tuning. However, as the number of quantization bits decreases, these conventional methods often result in a significant degradation of model accuracy. In response to this challenge, we introduce a novel non-uniform quantization approach that meticulously accounts for the influence of activation functions, outliers, and sensitive values on accuracy loss. This method effectively mitigates the accuracy degradation during the quantization process. Our experiments, conducted on the Llama-7B model, demonstrate that our technique achieves substantial compression with 2-bit quantization while preserving high model accuracy.
Authors: Dongmin Li, Bowen Chen, Dongyang Zhang
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4411747407