Model-free Reinforcement Learning for Continuous Time and State: A Stochastic Maximum Principle Approach
Year: 2026
Type: preprint
Abstract: This paper develops a model-free reinforcement learning (RL) algorithm based on the stochastic maximum principle for continuous-time stochastic control problems with continuous state and action spaces. For a parameterized Markovian policy, we establish the existence of the decoupling field for the adjoint backward stochastic differential equation, which allows the Hamiltonian gradient to be represented as a deterministic function of time and state. We then learn this Hamiltonian gradient directly from data and incorporate it into a policy gradient scheme with inexact gradients. We establish the convergence of the resulting policy gradient algorithm and establish the corresponding error estimates. Under suitable conditions on learning rates, exploration parameters, and approximation errors, the objective values converge and the $L^2$-norm of the policy gradient vanishes asymptotically. We further show that the proposed SMP-based policy gradient representation is equivalent to the existing continuous-time deterministic policy gradient representation based on the advantage-rate function. The effectiveness and efficiency of the proposed RL algorithm are demonstrated by numerical experiments.
Source: arXiv (Cornell University)
Authors: Bo, Lijun, Huang, Yijie, Wang, Jingfei
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7215032931