Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning
Year: 2023
Type: conference-paper
Abstract: This paper studies tabular reinforcement learning (RL) in the hybrid setting, which assumes access to both an offline dataset and online interactions with the unknown environment. A central question boils down to how to efficiently utilize online data collection to strengthen and complement the offline dataset and enable effective policy fine-tuning. Leveraging recent advances in reward-agnostic exploration and model-based offline RL, we design a three-stage hybrid RL algorithm that beats the best of both worlds -- pure offline RL and pure online RL -- in terms of sample complexities. The proposed algorithm does not require any reward information during data collection. Our theory is developed based on a new notion called single-policy partial concentrability, which captures the trade-off between distribution mismatch and miscoverage and guides the interplay between offline and online data.
DOI: 10.52202/075280-2426
Authors: Gen Li, Wenhao Zhan, Jason Lee, Yuejie Chi, Yuxin Chen
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G1056225131, G1469985387, Collaborative Research: CIF: Medium: MoDL:Toward a Mathematical Foundation of Deep Reinforcement Learning, G1523888516, RINGS: Learning-Enabled Ground and Air Integrated Networks (GAINs), G2384362821, CAREER: Towards a Theory of Deep Learning, Collaborative Research: CIF: Medium: Statistical and Algorithmic Foundations of Efficient Reinforcement Learning, G4720003262, Collaborative Research: CIF: Medium: Statistical and Algorithmic Foundations of Efficient Reinforcement Learning, CIF: Small: Taming Nonconvexity in High-Dimensional Statistical Estimation, G6098521345, CIF: Medium: Collaborative Research: Theory of Optimization Geometry and Algorithms for Neural Networks, G6671297155, G7452299184, G862988727, RI: Medium: Provable Reinforcement Learning with Function Approximation and Neural Networks, G8876996369, Collaborative Research: Towards a Theoretic Foundation for Optimal Deep Graph Learning
OpenAlex ID: W7133221461