Work

Foresight-Aware Reinforcement Learning for Infinite-Horizon Optimal Tracking Control

Year: 2026

Type: article

Abstract: Conventional reinforcement learning (RL) algorithms for the infinite-horizon optimal tracking suffer from short-sightedness, as their policy typically considers only the immediate reference state. This inherent limitation can lead to suboptimal tracking performance, particularly when tracking complex and dynamic trajectories. This article proposes a foresight-aware RL method that incorporates a multistep reference preview to overcome this limitation. We introduce the reference prediction model to infer future reference information based on$N$-step reference preview. We then construct an augmented tracking system that takes the system state and$N$-step reference signals as the augmented state. By proving that this augmented system satisfies the Markov property, we formulate a corresponding Bellman equation, which enables the learning of a stationary, time-invariant policy with foresight. Rigorous sensitivity analysis substantiates this approach, demonstrating that the optimal control’s dependence on future references decays exponentially. We further derive a performance bound, proving that the optimality gap diminishes exponentially as$N$increases, which theoretically validates the finite-preview strategy. We develop a model-based RL algorithm, foresight-aware approximate dynamic programming (FAADP), whose practicality is enabled by the theoretical finding that the policy gradient is independent of the reference prediction model’s derivatives. This key finding enables a novel training paradigm where future reference states are simply sampled from data, obviating the need for an analytical reference model. Simulation and real-world driving experiments on vehicle tracking demonstrate that as the preview horizon$N$increases, the proposed method significantly outperforms previous first-point tracking RL approaches.

Source: IEEE Transactions on Control Systems Technology

DOI: 10.1109/tcst.2026.3747302