APRES: Adaptive Policy-Aware Prioritized Experience Sharing for Scalable Decentralized Multi-Agent Reinforcement Learning
Year: 2026
Type: preprint
Abstract: Decentralized cooperative Multi-Agent Reinforcement Learning (MARL) faces sever non-stationarity and sample inefficiency, particularly in sparse-reward environments where informative experiences may be discovered by only a subset of agents. Experience sharing can improve exploration by allowing agents to reuse knowledge acquired by their peers; however, indiscriminate sharing may introduce redundant or conflicting information and become increasingly problematic as the number of agents grows. Existing approaches generally do not jointly consider (1) which experiences are useful to a receiver, (2) whom to learn from, and (3) When does sharing become more beneficial? To address this challenge, we present Adaptive Policy-aware Prioritized Experience Sharing (APRES), a lightweight and fully decentralized experience-sharing framework that addresses these three dimensions. First, APRES employs receiver-aware advantage prioritization, using the receiving agent's critic to identify shared experiences which are most informative relative to its current knowledge. Second, it introduces policy-aware weighting, based on clipped importance-sampling ratios to regulate the contribution of experiences generated by behaviorally different peers while limiting unstable gradient amplification. Third, an adaptive temporal decay schedule progressively reduces the contribution of shared experience as training advances, balancing early-stage collaborative exploration with late-stage policy specialization. Experiments on sparse-reward cooperative MARL benchmarks show that our approach achieves higher performance while training on less shared experiences, improves sample efficiency, and scales significantly better than state-of-the-art experience-sharing baselines.
Source: Research Square
Authors: Arefeh Esmaili, Mehdy Roayaei
Institution: Tarbiat Modares University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: Iran National Science Foundation
OpenAlex ID: W7236046976
