MeMoTune: A Measure and Moment-Driven Fine-Tuning Framework for Quantized Large Language Models
Year: 2025
Type: conference-paper
Abstract: Quantizing large language models (LLMs) is essential for reducing memory and computational costs in natural language processing.Existing methods combine quantization with parameter-efficient fine-tuning but often fail to meet practical performance requirements.This paper introduces MeMoTune, a novel fine-tuning framework for quantized LLMs.By employing a measure and moment approach within a low-rank approximation framework in probability measure space, MeMoTune optimizes the objective function for superior fine-tuning results.The update process is further refined through scaled gradient, enhancing convergence efficiency and noise robustness.Experiments on tasks like text generation, summarization, and understanding show MeMoTune significantly outperforms state-of-the-art methods, e.g.fine-tuning Llama2-13B on GSM8K improves accuracy by 5.5%, while fine-tuning DeBERTaV3-base on CoLA of GLUE increases Matthews correlation by 1.7%.The code is publicly available at github.com/hddyyyb/MeMoTune. LoRA AdapterC. Eliza′s overtime pay is $10 x 1.2 = $<<10*1.2=12>>12.Her total earnings for the first 40 hours is $10 x 40 = $<<10*40=400>>400.Her total overtime pay is $12 x 5 = $<<12*5=60>>60.Therefore, Eliza′s total earnings for this week is $400 + $60 = $<<400+60=460>>460.The final answer is: 460 Ground Truth: 460.0 C. weekly rate of Eliza for the first 40 hours of work is $10 x 40 = $<<10*40=400>>400.Eliza′s overtime pay is $10 x 1.2 = $<<10*1.2=12>>12.Her total earnings for the week are $400 + $12 = $<<400+12=412>>412.The final answer is: 412 Ground Truth: 460.0 B. The languages which use the Latin alphabet are English, Spanish, French, German, Italian, Portuguese, Dutch, Swedish, Danish, Norwegian, Finnish, Icelandic, Faroese, and Hungarian. B.The languages which use the Latin alphabet are English, Spanish, French, German, Portuguese, Italian, Dutch, Danish, Swedish, Norwegian, Finnish, and Icelandic.A. EVs are eco-friendly.A. EVs are better for the environment.
Authors: Yun Zhang, Xue Geng, Lizi Liao, Jintong Sun, Minghe Yu, Ge Yu
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Awards: G4466500611, G5051708620, G945082208
OpenAlex ID: W4412888765