Work

LLM Compression: How Far CanWe Go in Balancing Size and Performance?

Year: 2025

Type: conference-paper

Abstract: Quantization is an essential and popular technique for improving the accessibility of large language models (LLMs) by reducing memory usage and computational costs while maintaining performance.In this study, we apply 4-bit Group Scaling Quantization (GSQ) and Generative Pretrained Transformer Quantization (GPTQ) to LLaMA 1B, Qwen 0.5B, and PHI 1.5B, evaluating their impact across multiple NLP tasks.We benchmark these models on MS MARCO (Information Retrieval), BoolQ (Boolean Question Answering), and GSM8K (Mathematical Reasoning) datasets, assessing both accuracy and efficiency across various tasks.The study measures the tradeoffs between model compression and task performance, analyzing key evaluation metrics namely: accuracy, inference latency, and throughput (total output tokens generated per second), providing insights into the suitability of low-bit quantization for real-world deployment.Using the results, a user can then make a suitable decision based on the specifications that need to be met.We discuss the pros and cons of GSQ and GPTQ techniques on models of different sizes, which also serve as a benchmark for future experiments.

Source: International conference Recent advances in natural language processing

DOI: 10.26615/978-954-452-098-4-136