LLaMA-2 and LLaMA-3 Models with SparseGPT-Based Model Compression and LoRA/QLoRA Integration: Efficient Large Language Model Optimization for Resource-Constrained Environments
Year: 2025
Type: article
Abstract: Data exists in different sizes according to the needs and usage area. The need for these storage areas is increasing day by day. However, the increasing size of these models significantly increases storage requirements, memory usage, and inference latency, limiting applicability, especially on resourceconstrained devices (e.g., mobile, embedded systems). By 2025, global data volume is expected to reach 175 zettabytes, underscoring the growing need not only for efficient storage solutions but also for effective model optimization techniques [1]. Increasing model sizes and evolving computational requirements reveal the importance of model compression. The choice of compression algorithm plays a decisive role in balancing efficiency, speed, and accuracy. If the correct and efficient compression technique is chosen, both the compression ratio can be increased and the processing time can be reduced, while choosing an algorithm that is not suitable for the purpose can distract us from the target. When llama-2 and llama-3, which are among the large language models, are examined, the fact that they have millions of parameters can cause serious problems, especially in devices with limited storage and performance. The appropriate compaction technique can be selected by taking into account the characteristics as well as the overall performance metrics and size. With the SparseGPT used in this study, weights that can be considered unnecessary are pruned and the model is reduced. Using the SparseGPT-based pruning method, up to 5060% sparsity was applied on the LLaMA-2 and LLaMA-3 models, and significant memory savings were achieved without the need for retraining by setting the insignificant weights in the parameter space of the model to zero. This approach reduces the model size while eliminating the costly requirement of full retraining. Then the model is customized with lora/qlora. Instead of completely retraining large models, additional adjustments are made through adapters.
Authors: Arıbaş, Erke, Güler, Melisa
Institution: Istanbul Technical University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: diamond
OpenAlex ID: W7217744112