BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agents

Stepwise group-based reinforcement learning (RL) is a popular approach for training long-horizon LLM agents without a learned critic. However, these methods rely on grouping rollout steps to estimate advantages, assuming that steps within a group are interchangeable. The authors identify a fundamental 'state-action credit mismatch' in existing methods like GiGPO: observation-based partitioning is often too strict (creating singleton groups with no signal) or too coarse (failing to distinguish between different actions taken from the same state).

The authors introduce BiPACE (Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation). It replaces the standard observation-hash partition with a behavioral partition derived from the actor's own hidden-state geometry (BiGPO). This allows the model to group states that are behaviorally similar even if their surface-level observations differ. Within these clusters, the authors implement PACE, which uses action-conditioned peer baselines to estimate a local Q(s,a) - V(s) advantage. This allows the model to assign credit to specific actions rather than just the state-level return.

BiPACE significantly outperforms existing critic-free baselines across multiple benchmarks, including ALFWorld, WebShop, and TextCraft. On ALFWorld with Qwen2.5-7B, BiPACE_Q achieved a 97.1% validation success rate, compared to 90.8% for GiGPO. The authors demonstrate that their approach substantially reduces the 'singleton fraction'—the proportion of steps that provide zero gradient signal—by creating more meaningful, behaviorally coherent clusters. The computational overhead is minimal, measured at approximately 11.3% of a single training-step wall time.

This work addresses a critical bottleneck in training LLM agents: the difficulty of assigning credit for sparse, long-horizon rewards. By showing that the actor's own hidden-state geometry can serve as an effective proxy for bisimulation, the authors provide a way to improve sample efficiency and performance without the need for additional learned components like value networks or auxiliary losses. This makes the method a highly practical, drop-in improvement for existing agentic RL pipelines.