GSQ-Tuning: Group-Shared Exponents Integer in Fully Quantized Training for LLMs On-Device Fine-tuning
Year: 2025
Type: conference-paper
Abstract: Large Language Models (LLMs) fine-tuning technologies have achieved remarkable results.However, traditional LLM fine-tuning approaches face significant challenges: they require large Floating Point (FP) computation, raising privacy concerns when handling sensitive data, and are impractical for resourceconstrained edge devices.While Parameter-Efficient Fine-Tuning (PEFT) techniques reduce trainable parameters, their reliance on floating-point arithmetic creates fundamental incompatibilities with edge hardware.In this work, we introduce a novel framework for ondevice LLM fine-tuning that eliminates the need for floating-point operations in both inference and training, named GSQ-Tuning.At its core is the Group-Shared Exponents Integer format, which efficiently represents model parameters in integer format using shared exponents among parameter groups.When combined with LoRA-like adapters, this enables fully integer-based fine-tuning that is both memory and compute efficient.We demonstrate that our approach achieves accuracy comparable to FP16-based fine-tuning while significantly reducing memory usage (∼ 50%).Moreover, compared to FP8, our method can reduce ∼ 5 × power consumption and ∼ 11 × chip area with same performance, making large-scale model adaptation feasible on edge devices.
Authors: Sifan Zhou, Shuo Wang, Zhihang Yuan, Mingjia Shi, Yuzhang Shang, Dawei Yang
Institutions: Southeast University, Illinois Institute of Technology
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W4412887767