Year: 2025
Type: conference-paper
Abstract: Fine-Tuning of LLMs using RLHF / RLAIF has been shown as a critical step to improve the performance of LLMs in complex generation tasks.These methods typically use responselevel human or model feedback for alignment.Recent works indicate that finer sentence or span-level labels provide more accurate and interpretable feedback for LLM optimization.In this work, we propose FRACTAL, a suite of models to disaggregate responselevel labels into sentence-level (pseudo-)labels through Multiple Instance Learning (MIL) and Learning from Label Proportions (LLP) formulations, novel usage of prior information, and maximum likelihood calibration.We perform close to 2000 experiments across 6 datasets and 4 tasks that show that FRACTAL can reach up to 93% of the performance of the fully supervised baseline while requiring only around 10% of the gold labels.Furthermore, in a downstream eval, employing steplevel pseudo scores in RLHF for a math reasoning task leads to 5% absolute improvement in performance.Our work is the first to develop response-level feedback to sentence-level scoring techniques leveraging sentence-level prior information, along with comprehensive evaluations on multiple tasks as well as end-to-end finetuning evaluations.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W4412945040