Year: 2025
Type: conference-paper
Abstract: Fine-tuning large diffusion models for custom applications demands substantial power and time, which poses significant challenges for efficient implementation on mobile devices. In this paper, we develop a novel training accelerator specifically for Low-Rank Adaptation (LoRA) of diffusion models, aiming to streamline the process and reduce computational complexity. By leveraging a fully quantized training scheme for LoRA fine-tuning, we achieve substantial reductions in memory usage and power consumption while maintaining high model fidelity. The proposed accelerator features flexible dataflow, enabling high utilization for irregular and variable tensor shapes during the LoRA process. Experimental results show up to 1.81× training speedup and 5.50× energy efficiency improvements compared to the baseline, with minimal impact on image generation quality.
Institutions: Nanjing University, Sun Yat-sen University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4411724830