Rank-HIQL: Temporal Ranking for Hierarchical Offline Goal-Conditioned Reinforcement Learning
Year: 2026
Type: preprint
Abstract: Offline goal-conditioned reinforcement learning requires useful control signals for goals that may be far from the current state. We study whether temporal order within an observed trajectory can supply such a signal without bootstrapped value targets. Rank-HIQL replaces the value objective in HIQL with a two-head pairwise hinge loss over hindsight state–goal triples, while retaining its latent subgoal representation and hierarchical advantage-weighted policy extraction. We characterize the feasibility and scale ambiguity of this ordinal objective and show why accurate ranking alone does not guarantee effective policy extraction. On OGBench AntMaze Large, a held-out three-seed confirmation achieves 91.69% ± 0.87% success, whereas a paired PointMaze Large study exposes substantial late-training degradation despite nearly stable temporal-order accuracy. A frozen-value continuation and a local goal-relabeling intervention further separate critic drift from limitations of action supervision, identifying both the practical promise of temporal ranking and the conditions that must be addressed for reliable control.
Author: Xi, Zhihang
Institution: Southeast University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7225408938