Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes
Year: 2026
Type: preprint
Abstract: The average reward is a fundamental performance metric in reinforcement learning (RL) focusing on the long-run performance of an agent. Differential temporal difference (TD) learning algorithms are a major advance for average reward RL as they provide an efficient online method to learn the value functions associated with the average reward in both on-policy and off-policy settings. However, existing convergence guarantees require a local clock in learning rates tied to state visit counts, which practitioners do not use and does not extend beyond tabular settings. We address this limitation by proving the almost sure convergence of on-policy $n$-step differential TD for any $n$ using standard diminishing learning rates without a local clock. We then derive three sufficient conditions under which off-policy $n$-step differential TD also converges without a local clock. These results strengthen the theoretical foundations of differential TD and bring its convergence analysis closer to practical implementations.
Source: arXiv (Cornell University)
Authors: Blaser, Ethan, Wang, Jiuqi, Zhang, Shangtong
Institution: University of Virginia
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Awards: SLES: CRASH - Challenging Reinforcement-learning based Adversarial scenarios for Safety Hardening, Graduate Research Fellowship Program (GRFP), CAREER: Revolutionizing the Evaluation of AI Agents with Online and Offline Data, G6671297155, III: Small: Moving offline learning to rank online, from theory to practice
OpenAlex ID: W7130593181
Cites:
Cited by:
Related to: