A counterexample and a corrective to the vector extension of the Bellman equations of a Markov decision process
Year: 2024
Type: article
Abstract: Under the expected total reward criterion, the optimal value of a finite-horizon Markov decision process can be determined by solving the Bellman equations. The equations were extended by White to processes with vector rewards. Using a counterexample, we show that the assumptions underlying this extension fail to guarantee its validity. Analysis of the counterexample enables us to articulate a sufficient condition for White’s functional equations to be valid. The condition is shown to be true when the policy space has been refined to include a special class of non-Markovian policies, when the dynamics of the model are deterministic, and when the decision making horizon does not exceed two time steps. The paper demonstrates that in general, the solutions to White’s equations are sets of Pareto efficient policy returns over the refined policy space. Our results are illustrated with an example.
Source: Annals of Operations Research
Author: Anas Mifrani
Institutions: Université de Toulouse, Institut de Mathématiques de Toulouse
Language: English
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $2,990
OpenAlex ID: W4405524758