Work

Adaptive nonstationary value iteration for discounted control of Piecewise Deterministic Markov Processes

Year: 2026

Type: article

Abstract: We study the adaptive infinite-horizon discounted control problem for Piecewise Deterministic Markov Processes (PDMPs) using a Nonstationary Value Iteration (NVI) scheme. PDMPs, as introduced by Davis (Markov models and optimization, monographs on statistics and applied probability, Chapman and Hall, London, 1993) evolve deterministically between random jumps whose jump rate $$\lambda $$ λ , transition measure Q , and cost C depend on an unknown parameter $$\beta ^*$$ β ∗ . The proposed NVI algorithm recursively updates the value function using current parameter estimates, enabling online implementation. We show that, for any sequence of strongly consistent estimators $$\{\beta ^*_n\}$$ { β n ∗ } converging almost surely to $$\beta ^*$$ β ∗ , the resulting policy is asymptotically optimal under the discounted criterion.

Source: Mathematics of Control Signals and Systems

DOI: 10.1007/s00498-026-00447-x

Authors: O. L. V. Costa, F. Dufour, A. Genadot