Get started
Work

Weak-to-Strong Honesty Alignment via Group-Relative Policy Optimization

Year: 2026
Type: article
Abstract: Despite recent progress in aligning large language models (LLMs) with human intent, achieving honest alignment — models’ ability to express epistemic uncertainty and refuse to answer when lacking knowledge — remains an unsolved challenge, especially under low-resource supervision. In this paper, we introduce GREAT, a fundamentally novel alignment paradigm that integrates group-level reinforcement alignment with weak-to-strong epistemic supervision, explicitly targeting the honesty objective. Unlike prior methods that rely on costly human annotation or fragile pairwise preference modeling, our framework leverages a lightweight “honest head” trained with Group Relative Policy Optimization (GRPO) to identify high-fidelity uncertainty-aware responses from beam outputs, which are then used to bootstrap a large-scale pseudo-labeled corpus. Crucially, our design integrates (1) intra-group epistemic consistency constraints, (2) a dual-head actor-critic architecture, and (3) soft ensemble decoding to resolve the dilemma between cautious refusal and over-conservatism. Experiments across diverse domains — including opendomain QA and a sensitive military medical equipment testbed — demonstrate that GREAT not only outperforms prior alignment methods under limited data, but also generalizes robustly to unseen domains. Our framework introduces a new paradigm in LLM alignment by operationalizing epistemic humility as a trainable signal, offering a pathway toward safe and transparent deployment of LLMs in high-stakes settings. Extensive experiments show GREAT significantly boosts honest alignment in large models even with limited labeled data. Our code is available at https://github.com/cynicalight/Weak-to-Strong-Honesty-Alignment-via-Group-Relative-Policy-Optimization.
Source: Mathematics
DOI: 10.3390/math14030503
Institution: Wuhan University
Language: English
Open Access status: gold
Estimated APC paid: $3,137
OpenAlex ID: W7126200827
FWCI:
Cites:
Cited by:
Related to: