Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learning

In multi-task agentic reinforcement learning (RL), training a single policy to handle diverse tasks often leads to optimization instability. The authors investigate why shared-parameter models struggle with heterogeneous tasks, identifying a phenomenon they term "pace mismatch," where tasks exhibit divergent exploration-exploitation dynamics that lead to premature convergence or late-stage instability.

The authors analyze the entropy dynamics of agents during multi-task training and observe that tasks often experience "entropy crossovers" and spikes when using standard Group Relative Policy Optimization (GRPO). To address this, they propose Entropy Pacing Policy Optimization (EPPO). EPPO replaces the global, fixed clipping range in GRPO with a task-wise dynamic clipping mechanism. This mechanism uses an entropy-aware adaptive bound that tightens the update constraint for tasks that have collapsed to low-entropy (over-confident) states and relaxes it for tasks that remain in high-entropy (under-explored) states. Additionally, the method includes a stability-aware trend constraint to dampen oscillations when entropy increases.

EPPO effectively stabilizes multi-task training by organizing task entropies into more consistent, persistent bands. Quantitative diagnostics show that EPPO significantly reduces the frequency of inter-task entropy crossovers and suppresses late-stage entropy spikes compared to standard GRPO and other adaptive clipping baselines. Experimental results on five diverse agentic benchmarks (OS, DB, KG, ALFWorld, and WebShop) demonstrate that EPPO achieves higher average success rates than existing multi-task RL methods, suggesting that coordinating learning pace is critical for generalist agent performance.