Work

Unified algorithms for RL with Decision-Estimation Coefficients: PAC, reward-free, preference-based learning and beyond

Year: 2025

Type: article

Abstract: Modern Reinforcement Learning (RL) is more than just learning the optimal policy; alternative learning goals such as exploring the environment, estimating the underlying model and learning from preference feedback are all of practical importance. While provably sample-efficient algorithms for each specific goal have been proposed, these algorithms often depend strongly on the particular learning goal, and thus admit different structures correspondingly. It is an urging open question whether these learning goals can rather be tackled by a single unified algorithm. We make progress on this question by developing a unified algorithm framework for a large class of learning goals, building on the Decision-Estimation Coefficient (DEC) framework. Our framework handles many learning goals such as no-regret RL, PAC RL, reward-free learning, model estimation and preference-based learning, all by simply instantiating the same generic complexity measure called “Generalized DEC,” and a corresponding generic algorithm. The generalized DEC also yields a sample complexity lower bound for each specific learning goal. As applications, we propose “decouplable representation” as a natural sufficient condition for bounding generalized DECs, and use it to obtain many new sample-efficient results (and recover existing results) for a wide range of learning goals and problem classes as direct corollaries. Finally, as a connection, we reanalyze two existing optimistic model-based algorithms based on posterior sampling and maximum likelihood estimation, showing that they enjoy sample complexity bounds under similar structural conditions as the DEC.

Source: The Annals of Statistics

DOI: 10.1214/24-aos2483

Authors: Fan Chen, Song Mei, Yu Bai

Institutions: Massachusetts Institute of Technology, University of California, Berkeley, Salesforce (United States)

Language: English

Topic: Reinforcement Learning in Robotics

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: bronze

Funders: National Science Foundation, Office of Naval Research

Awards: CIF: SMALL: Theoretical Foundations of Partially Observable Reinforcement Learning: Minimax Sample Complexity and Provably Efficient Algorithms, CAREER: Theoretical foundations for deep learning and large-scale AI models, Mean Field Asymptotics in Statistical Inference: Variational Approach, Multiple Testing, and Predictive Inference, G8876996369