Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents

In long-horizon, sparse-reward tasks, LLM agents often receive a single terminal reward (success or failure) after many steps. Existing group-based reinforcement learning (RL) methods typically assign the same trajectory-level credit to every step, meaning that even locally correct actions in a failed trajectory are penalized. This paper investigates how to recover this "wasted" progress without the computational overhead of training a separate critic or reward model.

SCPO introduces a reward-shaping plugin that operates on rollout groups. For each group, it identifies a successful trajectory to serve as a reference. It then uses a frozen cross-encoder to compare the steps of failed trajectories against the successful reference. If a failed step demonstrates progress that matches a part of the successful reference not yet credited, SCPO assigns it positive step-level credit. This process is monotonic, ensuring that credit is granted only for new, unique progress, and it is integrated directly into the advantage estimation of existing group-based RL frameworks like GiGPO.

SCPO significantly improves performance on challenging agentic benchmarks, particularly for smaller models. On ALFWorld and WebShop, SCPO consistently matches or outperforms strong baselines like GiGPO, HCAPO, and HGPO. The gains are most pronounced in complex, multi-step tasks where the agent must recover from early mistakes. By reweighting the policy gradient toward failed steps that make meaningful semantic progress, SCPO achieves higher success rates without requiring additional rollouts, demonstrations, or learned verifiers.

This approach addresses the "semantic credit inconsistency" problem, where near-identical actions receive conflicting gradients based solely on the final outcome of their trajectory. By enabling agents to learn from their own partially successful attempts, SCPO provides a computationally efficient way to improve agentic reasoning in sparse-reward environments, making it a highly practical tool for post-training LLM agents.