Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attacks
Year: 2025
Type: preprint
Abstract: While Finetuning-as-a-Service (FaaS) enables customization of Large Language Models (LLMs) using user data, this service is vulnerable to safety degradation when user data includes harmful prompts, a threat known as harmful finetuning attacks. To defend against this, prior work first constructs safety-aligned LLM and then finetunes the LLM on user data. However, we observe that the safety-aligned weights provide weak initialization for downstream task learning, leading to suboptimal safety and utility. Motivated by this limitation, we shift the safe FaaS finetuning paradigm from finetuning safety-aligned weights to finetuning base weights under explicit safety-teacher guidance. Specifically, we propose a Refusal-Teacher (Ref-Teacher)-guided finetuning framework. Our approach directly finetunes the base LLM under the guidance of a safety-aligned Ref-Teacher, which filters harmful prompts from user data and distills safety into the base LLM during finetuning. Extensive experiments demonstrate that our paradigm effectively reduces harmful outputs while improving finetuning performance on user-specific tasks.
Source: arXiv (Cornell University)
Institution: Korea Advanced Institute of Science and Technology
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4417137532
Cites:
Cited by:
Related to: