S3: Stable Subgoal Selection by Constraining Uncertainty of Coarse Dynamics in Hierarchical Reinforcement Learning

In Hierarchical Reinforcement Learning (HRL), the high-level agent (Manager) often struggles with sparse, delayed feedback and the non-stationarity caused by a simultaneously learning low-level agent (Worker). The authors investigate whether providing the Manager with dynamics-aware intrinsic motivation—specifically, rewarding subgoals that the Worker can reach with high predictive certainty—can stabilize the hierarchy and improve sample efficiency.

The authors propose Stable Subgoal Selection (S3), which models the environment's coarse dynamics (transitions aggregated over multiple steps) using a Mixture Density Network (MDN). This network estimates the distribution of terminal states resulting from a given subgoal. The Manager receives an intrinsic reward proportional to the inverse of the dispersion (predictive uncertainty) of this distribution. By formulating this reward as a potential-based shaping function, the authors ensure that the intrinsic signal provides dense, capability-aware feedback without altering the optimal policy of the original Markov Decision Process.

Empirical evaluations in complex MuJoCo environments (Ant Fall, Ant Push, and Ant Maze) demonstrate that S3 consistently outperforms state-of-the-art HRL baselines like HIRO and HRAC. The results indicate that by steering the Manager toward subgoals that are reliably executable by the Worker, the framework reduces the impact of non-stationarity and enables more effective exploration in long-horizon tasks. The authors show that this approach is generalizable, as it can be integrated into existing HRL algorithms to enhance their stability.

This work addresses a fundamental bottleneck in HRL: the misalignment between the Manager's strategic planning and the Worker's evolving execution capabilities. By explicitly modeling the predictability of the hierarchy's coarse dynamics, S3 provides a principled way to bridge this gap, offering a robust alternative to methods that rely solely on reachability or novelty-based exploration.