Distributionally Safe Reinforcement Learning Under Model Uncertainty: A Single-Level Approach by Differentiable Convex Programming
Year: 2026
Type: article
Abstract: Safety assurance is uncompromisable in safety-critical environments, especially when drastic model uncertainties (e.g., distributional shift) exist, especially with humans in the loop. However, incorporating uncertainty in safe learning will naturally lead to a bi-level problem, where at the lower level, the worst-case safety constraint is evaluated within the uncertainty ambiguity set. In this paper, we present a tractable distributionally safe reinforcement learning framework that enforces safety under a distributional shift, as measured by a Wasserstein metric. To improve the tractability, we first use duality theory to transform the lower-level optimization from the infinite-dimensional probability space where distributional shift is measured, to a finite-dimensional parametric space. Moreover, by differentiable convex programming, the bi-level safe learning problem is further reduced to a single-level one with two sequential computationally efficient modules: a convex quadratic program to guarantee safety, followed by a projected gradient ascent to find the worst-case uncertainty simultaneously. This end-to-end differentiable framework with safety constraints offers a tractable single-level approach to addressing distributional safety. We test our approach on first- and second-order systems with varying complexities, including hardware demonstration on a 6-DOF drone. Compared with both uncertainty-agnostic policies and robust policies, our approach demonstrates a significant improvement in safety guarantees.
Source: Electronics
Authors: Alaa Eddine Chriat, Chuangchuang Sun
Institutions: Mississippi State University, Villanova University
Language: English
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $2,896
OpenAlex ID: W4387389825