Year: 2024
Type: preprint
Abstract: We study data-driven learning of robust stochastic control for infinite-horizon systems with potentially continuous state and action spaces. In many managerial settings--supply chains, finance, manufacturing, services, and dynamic games--the state-transition mechanism is determined by system design, while available data capture the distributional properties of the stochastic inputs from the environment. For modeling and computational tractability, a decision maker often adopts a Markov control model with i.i.d. environment inputs, which can render learned policies fragile to internal dependence or external perturbations. We introduce a distributionally robust stochastic control paradigm that promotes policy reliability by introducing adaptive adversarial perturbations to the environment input, while preserving the modeling, statistical, and computational tractability of the Markovian formulation. From a modeling perspective, we examine two adversary models--current-action-aware and current-action-unaware--leading to distinct dynamic behaviors and robust optimal policies. From a statistical learning perspective, we characterize optimal finite-sample minimax rates for uniform learning of the robust value function across a continuum of states under ambiguity sets defined by the $f_k$-divergence and Wasserstein distance. To efficiently compute the optimal robust policies, we further propose algorithms inspired by deep reinforcement learning methodologies. Finally, we demonstrate the applicability of the framework to real managerial problems.
Source: arXiv (Cornell University)
Authors: Wang, Shengbo, Meng, Jason, Si, Nian, Blanchet, Jose, Zhou, Zhengyuan
Institutions: Stanford University, New York University
Language: English
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: Collaborative Research: Safe Reinforcement Learning Guaranteed by Bayesian Distributionally Robust Optimization and Online Change Point Detection, Collaborative Research: CIF: Medium: Statistical and Algorithmic Foundations of Distributionally Robust Policy Learning, Collaborative Research: FET: Medium: Quantum Monte Carlo Speed Ups for Multilevel Computations and Other Statistical Algorithms, Collaborative Research: AMPS: Rare Events in Power Systems: Novel Mathematics, Statistics and Algorithms., G6098521345, Collaborative Research: CIF: Medium: Statistical and Algorithmic Foundations of Distributionally Robust Policy Learning
OpenAlex ID: W4399795014