Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Jie Wu, Zhengzhou Cai, Yueqing Sun, Ziang Ye, Linji Hao, Qi Gu, Xunliang Cai, Yongliang Shen, Yujiu Yang
6 min
Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks. Recent work introduces privileged self-distillation for credit assignment, providing denser supervision, but it remains unclear how such local signals should represent sequential credit. We propose AgentOPSD, a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space. This yields a principled reweighting scheme that converts sparse outcome supervision into turn-level credit signals and identifies pivotal turns through the marginal belief revision between consecutive states. The method is fully compatible with standard policy optimization and requires neither an additional critic nor extra rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B). AgentOPSD outperforms GRPO and strong self-distillation baselines, achieving 89.1% success on ALFWorld with Qwen2.5-7B. Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.
In long-horizon agentic tasks, reinforcement learning (RL) often relies on sparse, trajectory-level rewards. Standard methods like Group-Relative Policy Optimization (GRPO) broadcast this single reward uniformly across all actions in a sequence. This approach fails to distinguish between pivotal decisions that drive success and routine or redundant actions. The authors investigate how to derive more granular, turn-level credit signals to improve the efficiency and effectiveness of agentic training.
AgentOPSD (Recursive Self-Distillation for Agentic Reinforcement Learning) introduces a critic-free method for credit assignment. Instead of treating each turn in isolation, it interprets the discrepancy between a student policy and a privileged, skill-conditioned teacher as evidence. This evidence is aggregated at the turn level and used to recursively update a Bayesian belief state in log-odds space. By calculating the marginal revision of this belief state at each turn, the method generates a principled reweighting scheme that assigns higher credit to turns that significantly alter the probability of success.
AgentOPSD consistently outperforms GRPO and existing self-distillation baselines across three diverse benchmarks: ALFWorld, WebShop, and Search-QA. Notably, the method achieves an 89.1% success rate on ALFWorld using the Qwen2.5-7B model. Ablation studies confirm that the recursive nature of the belief updates and the alignment of credit with environment-defined turn boundaries are critical to these performance gains. Furthermore, the method demonstrates superior robustness to increasing task horizons compared to uniform-credit baselines, which suffer more significantly as the number of turns increases.
This work provides a scalable, critic-free solution to the credit assignment problem in long-horizon agentic RL. By transforming sparse outcome signals into history-dependent, turn-level feedback, AgentOPSD allows models to learn more effectively from complex, multi-step interactions without the overhead of additional rollouts or complex value-function approximations.
Alex: And by doing that at every single turn, you build up a detailed history of which moves were genuinely pivotal?
Sam: Right. The process runs recursively—meaning it feeds back on itself. The agent isn't just looking at one step in isolation. It's asking: how did this step change the overall probability of winning, given everything that happened before it? That history-dependent view is what makes the credit assignment much more precise than a simple blanket reward at the end.
Alex: That does sound considerably more informative than just saying "you won, so everything you did was good."
Sam: It is. The technical term for the signal that tells the model which actions to repeat is the "advantage." By reshaping that advantage using this history of belief updates, AgentOPSD focuses the learning on the specific, pivotal decisions that actually drove the outcome—rather than spreading credit evenly across every step.
Alex: That raises a concern for me, though. If the system is being that selective about which actions get credit, doesn't it risk becoming too sensitive to noise? Like, over-reacting to random flukes?
Sam: That's a valid concern, and the authors address it directly. They apply what they call a bounded advantage—essentially a safety governor on the learning signal. It caps how strongly the model can react to any single update, which prevents it from dramatically over-correcting based on a noisy or misleading signal from the teacher branch. It keeps the learning process stable.
Alex: So it's a deliberate trade-off: you gain precision in credit assignment, but you also build in a guardrail to prevent the model from chasing outliers.
Sam: Exactly. And that guardrail connects to a broader limitation the authors are candid about. The whole system depends on the quality of that teacher branch. If the teacher model is flawed, or if the initial estimate of success probability is off, the recursive updates can drift in the wrong direction. It's like learning from a tutor who occasionally gives the wrong answer—over time, those errors compound.
Alex: So the ceiling on the agent's performance is really set by how good that teacher version is during training.
Sam: That's a fair summary. The bounded advantage mitigates the problem, but it doesn't eliminate the dependency. The system is still fundamentally constrained by the information available to the teacher branch. That said, the authors suggest this mechanism opens a door for future work—agents could potentially use this same recursive credit assignment during deployment, not just training, to reflect on their own reasoning and improve long-term planning without needing an external reward at all.
Alex: So the immediate contribution is more reliable training, but the longer-term implication is agents that can audit their own decision-making. That's a meaningful shift in how we think about building these systems.
Sam: It is. By moving from simple outcome-based rewards to this kind of history-dependent credit assignment, the research suggests we gain considerably more control over what an agent actually learns from its experience. Thanks for listening to ResearchPod.