Year: 2025
Type: preprint
Abstract: From a first-principles perspective, it may seem odd that the strongest results in foundation model fine-tuning (FT) are achieved via a relatively complex, two-stage training procedure. Specifically, one first trains a reward model (RM) on some dataset (e.g., human preferences) before using it to provide online feedback as part of a downstream reinforcement learning (RL) procedure, rather than directly optimizing the policy parameters on said dataset via offline maximum likelihood estimation. In fact, from an information-theoretic perspective, we can only lose information via passing through a reward model and cannot create any new information via on-policy sampling. To explain this discrepancy, we scrutinize several hypotheses on the value of RL in FT through both theoretical and empirical lenses. Of the hypotheses considered, we find the most support for the explanation that on problems with a generation-verification gap, (1) it is relatively easy to learn the relatively simple RM (verifier) from the preference data. Then, (2) the downstream RL procedure only returns policies (generators) that are optimal for such relatively simple verifiers. Thus, end-to-end, two-stage online FT only has to search over a reduced subset of the full space of policies, requiring less data than offline FT.
Source: arXiv (Cornell University)
Institutions: Carnegie Mellon University, Cornell University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: RI: Small: Towards Provably Efficient Representation Learning in Reinforcement Learning via Rich Function Approximation, Collaborative Research: RI: Medium: Superhuman Imitation Learning from Heterogeneous Demonstrations, Collaborative Research: Inverse Task Planning from Few-Shot Vision Language Demonstrations, CAREER: Towards Real-world Reinforcement Learning
OpenAlex ID: W4415084114