Year: 2024
Type: dissertation
Abstract: Recent advancements in large language models (LLMs) have demonstrated remarkable capabilities in natural language processing tasks.However, their substantial size often poses significant challenges for deployment on resource constrained devices.This necessitates the development of efficient model compression techniques, with quantization emerging as a leading solution.While quantization can reduce model size and improve computational efficiency, its impact on the performance and accuracy of large language models remains a critical question.This thesis investigates the performance of eight large language models across various quantization levels, focusing on key tasks such as the MMLU-Pro for knowledge and reasoning, CRUXEval for code comprehension, and MuSR for reading comprehension and coherence.The results reveal a consistent trend: higher bit precision (e.g., 8-bit Q8_0) generally yields better performance, although the improvements diminish with further increases in precision.Contrarily, lower bit precision (e.g., 2-bit Q2_K) show lower but acceptable performance, but there are instances where substantial performance and coherence losses occur in some models.The findings indicate that while lower bit precision typically reduces performance, the impact varies significantly across different models and tasks.Larger models exhibit greater resilience to aggressive quantization, though they can still suffer significant performance drops at very low precision levels.Mid-sized models offer an optimal balance between capability, efficiency, and resource usage.These results provide valuable insights into the trade-offs between model size, quantization, and task performance, offering practical guidance for real-world applications.
Author: Baki Davidsson, Robin
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W6986681499