Imagine to Ensure Safety in Hierarchical Reinforcement Learning

How can reinforcement learning agents effectively navigate long-horizon tasks while strictly adhering to safety constraints? Existing flat-policy methods often struggle with the dual challenge of optimizing performance and safety over extended time horizons, leading to either poor task completion or frequent constraint violations.

The authors propose Imagine To Ensure Safety (ITES), a hierarchical reinforcement learning framework that decomposes the problem into two levels. A high-level policy generates intermediate subgoals, while a low-level policy executes actions to reach those subgoals. To ensure safety, the framework integrates two key components: a cost model that classifies states as safe or unsafe, and a world model that allows the agent to simulate potential future states. The high-level policy is trained to select subgoals that minimize expected costs, while the low-level policy uses the world model to perform 'imagined' rollouts, allowing it to evaluate the safety of its actions before executing them in the real environment.

ITES significantly outperforms existing state-of-the-art Safe RL baselines in complex, long-horizon robotic navigation and manipulation tasks. The framework demonstrates superior success rates while consistently maintaining safety constraints across different random seeds. By decoupling the safety optimization into hierarchical levels and using imagination to mitigate the impact of prediction errors, ITES effectively addresses the limitations of flat-policy architectures in high-dimensional, temporally extended scenarios.

This research provides a robust solution for deploying autonomous agents in real-world environments where safety is paramount. By demonstrating that hierarchical decomposition combined with model-based imagination can solve long-horizon tasks without sacrificing safety, the paper offers a scalable path forward for robotics and other safety-critical AI applications.