Directional Constraints for Efficient Exploration in Safe Reinforcement Learning

Safe Reinforcement Learning (SafeRL) often struggles with a trade-off between safety and performance. Standard safety layers, such as the Acting on the Tangent Space of the Constraint Manifold (ATACOM) framework, ensure safety by projecting actions into a safe tangent space. However, these methods are often overly conservative because they modify all actions near a constraint boundary—even those that are moving the agent away from danger. This unnecessary intervention restricts exploration and slows down the learning process.

To address this, the authors propose ATACOM Directional Constraints (ATACOM-DC). The core innovation is a mechanism that distinguishes between actions that approach a constraint and those that move away from it. By calculating the derivative of the constraint with respect to the action, the algorithm identifies which constraints are currently threatened. If an action is safe (i.e., it does not increase the constraint violation), the safety layer remains inactive, allowing the agent to explore freely. If an action is unsafe, the layer intervenes to project the action back into the safe set.

This approach effectively reduces the "conservative bias" inherent in many safety-constrained RL methods. By only applying constraints when strictly necessary, the agent can learn faster and achieve higher task performance in complex robotic environments. The authors demonstrate that this method maintains the same safety guarantees as the original ATACOM framework while significantly improving sample efficiency across several challenging robotic control tasks, including air hockey and quadrotor navigation.