Year: 2026
Type: conference-paper
Abstract: Large language models (LLMs) face a critical alignment challenge: balancing safety with helpfulness.Excessive safety can lead to overrefusal, where models reject harmful-looking yet benign queries, severely limiting utility.Existing training-free interventions offer an efficient way to mitigate over-refusal without retraining, but suffer from high inference overhead and architecture dependency.Our work explores a complementary direction: rather than applying post-hoc corrections to model outputs, our goal is to intrinsically reshape the distributions of harmful and benign samples within the model's decision space.In this paper, we argue that a lightweight trainingbased approach can more effectively distinguish between harmful and benign samples.We propose Single Token Alignment (STA), which optimizes only a single-token prefix (e.g., 4,096 parameters) while keeping the base model frozen.To address the inherent challenge of achieving robust refinement through such a minimal parameter interface, STA employs a mixed weighting mechanism integrated with its optimization objective.This mechanism incorporates hard weighting via stringent data filtering to provide clear, unbiased learning signals, and soft weighting through a focal mechanism to prioritize challenging cases.Extensive experiments across 9 models and 10 datasets demonstrate that STA achieves a superior safety-helpfulness balance for LLMs, MLLMs, and reasoning models, offering a highly efficient and generalizable solution for refining safety alignment.
Institutions: Wuhan University, LinkedIn (United States), University of Oxford
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Award: G3488128904
OpenAlex ID: W7166835435