A modular approach to enhancing safety in text-to-image diffusion models via targeted LoRA fine-tuning
Year: 2025
Type: conference-paper
Abstract: This paper discusses recent advances in text-to-image generative models, which improve image quality and prompt alignment but also introduce vulnerabilities to harmful content from adversarial prompts.Existing safety measures often fail against complex attacks.We propose a novel safety approach using targeted Low-Rank Adaptation (LoRA) fine-tuning combined with Metric learning.Our approach aimed for modification of latent vectors of harmful prompts, aligning them with safe content to reduce unsafe outputs.Experiments demonstrate safety levels comparable to industry benchmarks, particularly when adapters are trained on specific harm categories.Our study provides a reusable framework to protect against harmful outputs that can be scaled to protect against multiple prompt categories.
Authors: Maksym Kizitskyi, Oleksii Turuta
Institution: Kharkiv National University of Radio Electronics
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
OpenAlex ID: W7084089785
FWCI:
Cites:
Cited by:
Related to: