Year: 2023
Type: preprint
Abstract: Model-based methods have recently shown great potential for off-policy evaluation (OPE); offline trajectories induced by behavioral policies are fitted to transitions of Markov decision processes (MDPs), which are used to rollout simulated trajectories and estimate the performance of policies. Model-based OPE methods face two key challenges. First, as offline trajectories are usually fixed, they tend to cover limited state and action space. Second, the performance of model-based methods can be sensitive to the initialization of their parameters. In this work, we propose the variational latent branching model (VLBM) to learn the transition function of MDPs by formulating the environmental dynamics as a compact latent space, from which the next states and rewards are then sampled. Specifically, VLBM leverages and extends the variational inference framework with the recurrent state alignment (RSA), which is designed to capture as much information underlying the limited training data, by smoothing out the information flow between the variational (encoding) and generative (decoding) part of VLBM. Moreover, we also introduce the branching architecture to improve the model's robustness against randomly initialized model weights. The effectiveness of the VLBM is evaluated on the deep OPE (DOPE) benchmark, from which the training trajectories are designed to result in varied coverage of the state-action space. We show that the VLBM outperforms existing state-of-the-art OPE methods in general.
Source: arXiv (Cornell University)
Authors: Gao, Qitong, Gao, Ge, Chi, Min, Pajic, Miroslav
Institutions: Duke University, North Carolina State University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: G1257962346, G1523888516, CAREER: Foundations for Secure Control of Cyber-Physical Systems, Generalizing Data-Driven Technologies to Improve Individualized STEM Instruction by Intelligent Tutors, Integrated Data-driven Technologies for Individualized Instruction in STEM Learning Environments, CAREER: Improving Adaptive Decision Making in Interactive Learning Environments, AI Institute for Edge Computing Leveraging Next Generation Networks (Athena), CPS: Medium: Collaborative Research: Human-on-the-Loop Control for Smart Ultrasound Imaging
OpenAlex ID: W4389852026