Work

Fundamental safety-capability trade-offs in fine-tuning large language models

Year: 2026

Type: article

Abstract: Fine-tuning large language models (LLMs) on some task-specific datasets has been a primary use of LLMs. However, it has been empirically observed that this approach to enhancing capability inevitably compromises safety, a phenomenon also known as the safety-capability trade-off in LLM fine-tuning. This article presents a theoretical framework for understanding the interplay between safety and capability in two primary safety-aware LLM fine-tuning strategies, providing new insights into the effects of data similarity, context overlap, and alignment loss landscape. Our theoretical results characterize the fundamental limits of the safety-capability trade-off in LLM fine-tuning, which are also validated by numerical experiments.

Source: PNAS Nexus

DOI: 10.1093/pnasnexus/pgag097

Authors: Pin-Yu Chen, Han Shen, Payel Das, Tianyi Chen

Institutions: IBM (United States), Rensselaer Polytechnic Institute, Cornell University, Cornell Tech

Language: English

Topic: Adversarial Robustness in Machine Learning

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: gold

Estimated APC paid: $4,202

Funder: National Science Foundation