Year: 2026
Type: conference-paper
Abstract: Reinforcement learning (RL) in non-stationary environments presents significant challenges as distribution shifts in transition dynamics and rewards rapidly invalidate previously gathered experience. Traditional Experience replay (ER) methods that prioritize solely by temporal difference error (TD-error) conflate policy-induced distribution shift with exogenous environment changes, yielding suboptimal sampling and slow adaptation. To address this limitation, we introduce the Discrepancy of Environment (DoE), a metric that isolates the impact of environment transitions on value estimation. Building on DoE, we further propose the Discrepancy of Environment Prioritized Experience Replay (DEER), which jointly prioritizes samples by policy evolution and detected environmental variation. DEER employs a binary change-point detector to signal shifts and, conditional on detection, switches between pre- and post-shift prioritization tailored to the current dynamics. This design improves sample efficiency and stabilizes learning under dynamic conditions. Across four non-stationary benchmarks, DEER improves off-policy performance by 11.54% on average relative to the strongest ER baseline.
Authors: Tianyang Duan, Zongyuan Zhang, Songxiao Guo, Yuanye Zhao, Zheng Lin, Zihan Fang, Yi Liu, Dianxin Luan, Dong Huang, Heming Cui, Yong Cui
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4417077740
FWCI:
Cites:
Cited by:
Related to: