Get started
Work

Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluation

Year: 2026
Type: preprint
Abstract: In reinforcement learning policy evaluation, classic on-policy methods often suffer from high variance when estimating policy performance. To mitigate this issue, behavior policy search has been proposed to learn data-collecting policies tailored to reduce online evaluation variance. However, these approaches do not account for uncertainties in the transition functions. In practice, simulator transitions often differ from the real world due to modeling errors or approximation limitations. As a result, behavior policies trained in simulation may still yield high variance when deployed in real environments, leading to costly reliance on real-world evaluation samples. In this work, we propose a double-loop gradient-based algorithm for learning behavior policies that are both efficient and robust to transition uncertainty. Theoretically, we derive novel transition-variance gradient expressions and establish global convergence guarantees for the algorithm. Numerically, we demonstrate that our method is less sensitive to transition perturbations than existing approaches, providing supportive evidence for its practical utility.
DOI: 10.48550/arxiv.2608.24146
Authors: Chen, Claire, Liu, Shuze Daniel, Luo, Licheng, Chandra, Rohan, Jiang, Nan, Zhang, Shangtong
Institutions: California Institute of Technology, Purdue University West Lafayette, Massachusetts Institute of Technology, University of California, Riverside, University of Virginia, University of Illinois Urbana-Champaign
Open Access status: green
Awards: AI Institute for Future Edge Networks and Distributed Intelligence (AI-EDGE), SLES: CRASH - Challenging Reinforcement-learning based Adversarial scenarios for Safety Hardening, CAREER: Revolutionizing the Evaluation of AI Agents with Online and Offline Data, G6671297155, III: Small: Moving offline learning to rank online, from theory to practice, CAREER: Theoretical Foundations of Offline Reinforcement Learning
OpenAlex ID: W7204251088
Cites:
Cited by:
Related to: