Work

Safe reinforcement learning force control for a hydraulic actuator with real-world training using a learned contraction metric

Year: 2026
Type: article
Abstract: Force control in hydraulic actuators remains challenging because of strong nonlinearities, uncertainties, and the risks associated with unsafe exploration during learning. This work examines safe reinforcement learning (RL) for hydraulic force control with real-world training using contraction-metric certificates. A data-driven model of the hydraulic actuator, identified from experimental data, is used for simulation-based pre-training of Soft Actor-Critic (SAC) policies that adapt the gains of feedback-linearization (FL) and sliding-mode control (SMC) force controllers. To limit instability during online training, we propose a quadratic-programming (QP) contraction filter that uses a learned contraction metric to enforce an approximate contraction condition in the force-tracking direction while introducing minimal corrections to the policy output. The framework is validated on a hydraulic test bench, where RL policies are trained directly on hardware and benchmarked against fixed-gain and simulation-trained counterparts. The experimental results indicate that real-hardware training mitigates the gain over-optimization and instability observed in simulation-only policies, with the most evident benefits appearing for the SMC-based learning controller. The results further show that the proposed method does not consistently outperform well-tuned fixed-gain controllers in all operating conditions, especially when the nominal model is already accurate. These findings suggest that contraction-based certificates can support safe RL in high-force hydraulic systems, while also emphasizing the local and task-dependent nature of the learned safety certificate.
Language: English
Open Access status: gold
Estimated APC paid: $1,470
OpenAlex ID: W7128529314
FWCI:
Cites: 42
Cited by:
Related to: 10