Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning

Reinforcement learning (RL) agents often struggle in environments where rewards are sparse and the time horizon is long, as the probability of randomly encountering a reward is extremely low. This paper investigates whether a hierarchical reinforcement learning (HRL) architecture, which separates high-level strategic planning from low-level continuous control, can improve exploration and sample efficiency in these challenging settings.

The authors propose the Hierarchical Soft Actor-Critic (HRL-SAC) framework. This architecture uses a two-level hierarchy: a meta-controller (high-level) that selects sub-goals at a coarse temporal scale, and a low-level controller that executes primitive actions to achieve those sub-goals. Both levels utilize the Soft Actor-Critic (SAC) algorithm, which incorporates entropy regularization to encourage stochastic exploration and stabilize training. The low-level controller is guided by an intrinsic reward function that minimizes the distance between the current state and the assigned sub-goal, effectively bridging the gap between strategic planning and reactive execution.

The HRL-SAC framework was evaluated using the Search-and-Rescue-2 (SAR-2) dataset. The results demonstrate that the hierarchical approach significantly outperforms a flat SAC baseline in terms of success rates, coverage efficiency, and convergence speed. By decomposing the decision-making process, the agent effectively addresses the problem of delayed credit assignment, allowing it to learn optimal policies in environments where standard RL methods typically fail.

This work provides a robust, theoretically grounded solution for complex robotic tasks, such as search-and-rescue, where agents must navigate large spaces with limited feedback. By applying entropy regularization at multiple temporal levels, the framework offers a scalable way to handle continuous control in long-horizon tasks without relying on task-specific heuristics or complex manual tuning.