SD2AC: A reinforcement learning framework using distribution evaluation and sequential decision-making for UCAV combat
Year: 2025
Type: article
Abstract: The increasing complexity of autonomous drone swarm operations, particularly in adversarial air combat scenarios, presents significant challenges in multi-agent reinforcement learning (MARL). Existing approaches often face issues such as poor coordination, suboptimal convergence, and overestimation bias, undermining their applicability to high-stakes environments. We propose the Sequential Decision Distribution Soft Actor-Critic (SD2AC), a novel off-policy MARL framework for drone swarm coordination to address these challenges. SD2AC introduces three key innovations: (1) a sequential decision-making framework in which agents adaptively condition their decisions on the policies of preceding agents, enhancing coordination and reducing conflicts; (2) a distributional Q-value critic that models the full return distribution to mitigate overestimation bias and improve policy robustness; and (3) an adaptive twin value distribution learning mechanism that leverages dual critics to dynamically select conservative value estimates, ensuring stable learning under uncertainty. Rigorous evaluation in diverse environments, including Close Air Combat, Multi-agent Encirclement with Collision Avoidance (MECA), and MAMuJoCo, demonstrate the superiority of SD2AC over state-of-the-art MARL methods in terms of reward optimization, convergence speed, and coordination efficiency. Ablation studies further validate the individual contributions of each component.
DOI: 10.1093/jcde/qwaf045
Authors: Tao Yang, Xinhao Shi, Cheng Xu, Yulin Yang, Hongzhe Liu, Qinghan Zeng
Institution: Beijing Union University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,650
OpenAlex ID: W4409828465