DPLoRA: A Dual-Pruning Framework based on ILP Optimization and Progressive Pruning for Parameter-Efficient LoRA Fine-Tuning
Year: 2026
Type: conference-paper
Abstract: We propose DPLoRA (Dual-Pruning Low-Rank Adaptation), a framework that optimizes rank allocation via two stages: (1) an initial pruning stage (OPLoRA; Optimal Pruning LoRA) that uses Integer Linear Programming (ILP) to determine optimal layer-wise ranks without manual tuning; and (2) a progressive pruning stage that further reduces ranks adaptively during training using importance scores smoothed by Exponential Moving Average (EMA).Experiments demonstrate that OPLoRA consistently outperforms existing PEFT baselines on GLUE and instructionfollowing tasks, while the full DPLoRA framework establishes a new state-of-the-art among compared PEFT baselines on GLUE at highperformance settings (p = 0.4).At efficiencyfocused settings (p = 0.8), our method reduces trainable parameters by over 80% and training time by 46% compared to standard LoRA, offering a highly efficient solution for deploying large-scale models in resource-constrained environments.
Authors: Changjun Park, Sejong Yoon, Jaekwang Kim
Institutions: Sungkyunkwan University, The College of New Jersey
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Awards: G1428394818, G1977249509, G2514582570, G2652541173, G314908243, G3812690899, G4256929933, G4288452366, G440996017, G5093466393, G5593258703, Equipment: MRI: Track 1 Acquisition of Current Hardware to Enhance Computational Research on the ELSA High Performance Computing Cluster at The College of New Jersey, CC* Networking Infrastructure: Building a Friction-free High-speed Science Network and DMZ to Support Data-intensive Research and Education at The College of New Jersey, G6500653232
OpenAlex ID: W7166832160