LOTAPO: Leave-One-Turn Attribution for Policy Optimization with Self-Generated Process Rewards in Multi-Turn Search Reasoning

In multi-turn search reasoning, reinforcement learning (RL) models are typically trained using only terminal outcome rewards—signals that indicate whether the final answer is correct. This creates a credit-assignment problem: the model cannot distinguish between useful, redundant, or harmful intermediate search steps. The authors ask whether a model can generate its own process-level supervision to guide these intermediate steps without relying on external reward models, verifiers, or human-annotated data.

The authors introduce LOTAPO (Leave-One-Turn Attribution for Policy Optimization). For every search turn in a trajectory, the model performs a counterfactual experiment: it replaces the specific search action and its retrieved observation with a [DELETE] placeholder. It then measures the change in the policy's mean log-likelihood of the gold answer. This "Answer-Likelihood Gain" quantifies the contribution of that specific turn while keeping all other downstream interactions intact. To ensure stability, the method applies robust scaling and a "sign-consistency gate," which only applies a process reward if the normalized advantage aligns with the raw attribution score. This signal is then combined with the standard terminal outcome reward to update the policy.

LOTAPO demonstrates that retrospective, policy-derived attribution provides effective process supervision. Across seven knowledge-intensive question-answering datasets, LOTAPO achieved an average exact-match score of 0.326, outperforming the strongest step-reward baseline (IGPO) by 0.053. Ablation studies confirm that both the backward attribution mechanism and the sign-consistency gating are essential for performance, proving that the model can learn to prioritize helpful search turns and ignore redundant or harmful ones using only its own internal probabilistic estimates.