Does Policy Mirror Descent Converge in Policy with Non-Decomposable Mirror Maps? A Counterexample
Year: 2026
Type: preprint
Abstract: Policy mirror descent (PMD) refers to a class of policy optimization methods associated with different mirror maps. For a broad class of decomposable Legendre mirror maps encompassing many standard choices, Li and Wei [2026] show that the PMD policy sequence converges to an optimal policy from any admissible initialization under any positive constant step size, even when the optimal policy is not unique. In this note, we provide a counterexample with assistance from OpenAI's GPT-5.6 model, showing that PMD with a non-decomposable Legendre mirror map need not converge in policy.
Source: arXiv (Cornell University)
Authors: Li, Wenye, Wei, Ke
Institution: Fudan University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7222248234