ResearchPod Summary
In long-horizon agentic tasks, reinforcement learning (RL) often struggles with reward sparsity and the credit assignment problem. Existing step-level RL methods treat agent exploration as a collection of isolated, linear trajectories, which leads to high-variance state-value estimation and myopic, localized feedback. This paper asks how we can better leverage the inherent structure of multi-turn interactions to provide more accurate, fine-grained training signals for LLM agents.
The authors propose Group-Graph Policy Optimization (G2PO), which reconstructs sampled trajectories into a global state-transition graph. Instead of evaluating actions based on single-trajectory outcomes, G2PO introduces two primary innovations:
G2PO was evaluated on three representative long-horizon benchmarks: WebShop, ALFWorld, and AppWorld. The results demonstrate that G2PO consistently outperforms state-of-the-art prompt-based and RL baselines. Specifically, the method achieved success rate improvements of up to 22.2% over the GRPO baseline on WebShop and 14.4% on ALFWorld. These gains are achieved with minimal additional computational overhead, validating the efficiency of the graph-based approach.
This work addresses a fundamental limitation in current LLM agent training: the inability to distinguish between trivial actions and critical breakthroughs in long-horizon tasks. By shifting from a trajectory-centric view to a graph-centric view, G2PO provides a more robust framework for credit assignment, enabling agents to learn more effectively from sparse, delayed rewards without requiring expensive process-level annotations.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.