Year: 2024
Type: dissertation
Abstract: Large Language Models (LLMs) require a finetuning phase post general pre-training to align their outputs with human expectations, namely embodying helpfulness, harmlessness, and honesty. Prominent alignment techniques involve learning from human feedback, using Reinforcement Learning (RLHF). They use preference signals on binary comparisons between LLM generations for training a Reward Model that ultimately assigns a holistic scalar reward to generated outputs. However, human preferences are often nuanced and intricately articulated in natural language, surpassing binary constraints and numerical rewards. We develop an approach that can use language feedback to align an LLM. We propose a 3-step iterative algorithm leveraging Synthetic Language Feedback that alternates: (1) sampling LLM-generated text with the current policy, (2) providing synthetic language feedback to samples by mapping reward quantiles obtained through a Reward Model to feedback tags, (3) prepending the feedback to the LLM input prompt and learning the conditional probability distribution of generations given feedback and prompt with a standard conditional language modeling objective, while encouraging the LLM to preserve language fluency and diversity by employing a KL-divergence penalty that keeps the model close to the pre-trained LLM. By modeling the distribution of feedback and generations, the expected aligned behavior can be obtained at inference time by conditioning on feedback associated with a high-reward quantile. We achieved state-of-the-art performance in a single-reward unlearning toxicity task, achieving the lowest toxicity without compromising language quality as compared to any other previous work, and attained competitive results in a multi-reward Long-Form QA task, while relying only on language modeling primitives and no hyperparameter tuning. Our synthetic feedback experiments show that language-based feedback can be a viable alternative to contrastive feedback for aligning LLMs.
Author: Abadal Lloret, Saüc
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7010917313