Adaptive Multi-Horizon Reinforcement Learning

Reinforcement learning (RL) agents typically rely on a single, fixed discount factor to balance immediate rewards against long-term goals. However, biological agents demonstrate the ability to flexibly adjust their planning horizon based on the task. This paper investigates how an agent can dynamically adapt its planning horizon to different environmental demands, particularly in continual learning scenarios where reward structures change over time.

The authors introduce a mixture-of-experts framework where multiple action-value functions (critics) are learned in parallel, each using a different discount factor. A state-dependent gating network learns to assign weights to these critics, effectively creating a composite value function that adapts to the current context. The gating network is optimized using the TD error of the mixed value function, allowing the agent to prioritize different temporal horizons as it encounters new tasks or environmental configurations.

Empirical evaluations in MiniGrid environments demonstrate that the multi-horizon approach successfully identifies effective discount factors across diverse tasks, including foraging, goal-reaching, and complex multi-room navigation. In continual learning experiments, the agent maintains high performance by shifting its reliance between different critics as it transitions between tasks. The results suggest that this adaptive mechanism provides a robust alternative to manual discount-factor tuning and improves parameter efficiency in dynamic environments.

This work bridges the gap between fixed-horizon RL and the flexible, multi-timescale decision-making observed in biological systems. By enabling agents to automatically adjust their temporal perspective, this framework offers a promising path toward more autonomous and adaptable AI systems capable of handling the complexities of real-world, non-stationary environments.