Work

Reinforcement learning in convergently non-stationary environments: Feudal hierarchies and learned representations

Year: 2025

Type: article

Abstract: We study the convergence of Q -learning-based methods in convergently non-stationary environments, particularly in the context of hierarchical reinforcement learning and of dynamic features encountered in deep reinforcement learning. We demonstrate that Q -learning achieves convergence in tabular representations when applied to convergently non-stationary dynamics, such as the ones arising in a feudal hierarchical setting. Additionally, we establish convergence for Q -learning-based deep reinforcement learning methods with convergently non-stationary features, such as the ones arising in representation-based settings. Our findings offer theoretical support for the application of Q -learning in these complex scenarios and present methodologies for extending established theoretical results from standard cases to their convergently non-stationary counterparts.

Source: Artificial Intelligence

DOI: 10.1016/j.artint.2025.104382

Authors: Diogo S. Carvalho, Pedro A. Santos, Francisco S. Melo

Institutions: Instituto de Engenharia de Sistemas e Computadores Investigação e Desenvolvimento, University of Lisbon, Instituto Superior Técnico

Language: English

Topic: Reinforcement Learning in Robotics

Subfield: Artificial Intelligence

Field: Computer Science

Domain: Physical Sciences

Open Access status: hybrid

Estimated APC paid: $4,260

Funder: Fundação para a Ciência e a Tecnologia