Yijun Lu, Rui Ye, Jiajun Wang, Yuwen Du, Tian Jin, Songhua Liu, Siheng Chen
6 min
Long-horizon search agents must perform complex, multi-step investigations—such as formulating queries, inspecting evidence, and revising hypotheses—to reach a final answer. A major challenge in training these agents is the credit-assignment problem: existing methods typically treat entire search trajectories as uniformly successful or failed. This approach fails to distinguish between useful intermediate actions (even in failed attempts) and erroneous or redundant steps (even in successful ones), limiting the agent's ability to learn effective search behaviors.
The authors propose Answer-Backtracked Credit Assignment (ABC), a framework that provides fine-grained supervision for training search agents. The process consists of two main stages:
These dense, step-level rewards are then used to train the agent through two methods: ABC-SFT, which reweights the training loss of each turn based on its score, and ABC-GRPO, which incorporates the step-level scores as rewards within a Group Relative Policy Optimization (GRPO) framework.
The researchers trained ABSeeker, a 4B-parameter model based on Qwen3.5-4B, using the ABC framework. ABSeeker significantly outperforms other 4B-scale search agents and achieves performance levels comparable to much larger models (approximately 30B). Specifically, it reached 55.3% on the BrowseComp benchmark and 52.9% on BrowseComp-ZH when using context management. The results demonstrate that providing dense, step-level feedback is highly effective for optimizing the reasoning and search trajectories of long-horizon agents.
This work addresses a fundamental bottleneck in training autonomous agents for complex, multi-step tasks. By moving away from sparse, trajectory-level supervision, the authors provide a principled way to teach agents which specific actions contribute to success. This allows smaller, more efficient models to achieve high-level reasoning capabilities, making advanced search agents more accessible and computationally feasible.
Long-horizon search agents must make multiple sequential actions (steps) to search, retrieve, verify, and integrate evidence to reach a final answer. However, existing methods for training these agents typically treat all steps within a trajectory uniformly during both supervised fine-tuning (SFT) and reinforcement learning (RL), failing to distinguish useful actions from erroneous or redundant ones. In this paper, we propose Answer-Backtracked Credit Assignment (ABC), a fine-grained credit assignment framework for training long-horizon search agents by converting sparse trajectory-level outcomes into dense step-level supervision that rewards useful actions (even in failed trajectories) while suppressing erroneous or redundant actions. Specifically, given a potentially obscure query and its corresponding ground-truth answer, ABC first performs Answer-Backtracked Clue Recovery, which traces back from the answer to recover intermediate clues required to solve the question. It then applies Clue-Anchored Step Scoring to evaluate each search step against these clues, converting sparse binary outcome supervision into dense step-level rewards. Based on these rewards, we develop ABC-SFT, which reweights the loss of each turn, and ABC-GRPO, which uses the step-level scores as rewards in GRPO. Building on this framework, we train ABSeeker based on Qwen3.5-4B with only 8.5k examples. ABSeeker achieves 37.3% on BrowseComp and 39.1% on BrowseComp-ZH. With context management, the scores further improve to 55.3% and 52.9%, respectively, significantly outperforming same-scale (4B) agents and even matching the performance of larger ones (approximately 30B). These results demonstrate the effectiveness of answer-backtracked step-level credit assignment for training long-horizon search agents.
Alex: It does. The paper reports that a relatively small model — one with around four billion parameters — can outperform models that are roughly seven times larger when trained with this method.
Sam: That's a meaningful difference. So the quality of the feedback loop matters more than the raw size of the model?
Alex: That's the central argument. When you provide precise, step-by-step feedback, the model doesn't need to rely on sheer scale to perform well. It can learn more efficiently from less.
Sam: How do they actually turn those clue scores into something the model can learn from? What does the training look like in practice?
Alex: They use two methods in sequence. The first is called "ABC-SFT" — supervised fine-tuning with a twist. In standard fine-tuning, you show the model examples of correct behaviour and it tries to copy them, treating every part of the example equally.
Sam: Like reading a whole textbook chapter and hoping you absorb the important bits.
Alex: Right. ABC-SFT changes that by using the step-level scores to reweight the training data. It tells the model: focus your learning on these high-scoring steps. Ignore the noise in the less effective parts. It's more like a teacher highlighting the key sentences before you read.
Sam: And the second method?
Alex: That's "ABC-GRPO" — Group Relative Policy Optimisation. Rather than showing the model one correct example, it generates a whole group of different attempts at the same problem and compares them against each other.
Sam: So instead of saying "this is the right answer, copy it," you're saying "here are ten attempts — these three were better than the others, and here's why"?
Alex: Exactly. It uses the step-level scores to identify which attempts consistently hit the right clues, and nudges the model toward those strategies. It rewards genuine reasoning, not just a fortunate outcome.
Sam: So the two stages work together — first you teach it to imitate good steps, then you train it to prefer good strategies when it has to choose between options.
Alex: That's a good way to put it. It moves training from a blunt instrument — one final grade — to something more like a coaching process, where every decision along the way is evaluated.
Sam: But this whole approach depends on having a verified answer to work backward from. What happens when there isn't one clear correct answer?
Alex: That's the main limitation the authors acknowledge. The method requires a ground-truth answer to anchor the evidence chain. For open-ended or subjective tasks, where there's no single verifiable fact, it currently struggles.
Sam: So it's best suited to tasks like factual web search, where you can eventually confirm whether something is true or false.
Alex: Precisely. The paper notes that future work aims to use a secondary model — essentially a critic — to simulate that verification step, even when a clear answer isn't immediately available.
Sam: So the bigger picture here is that we've been thinking about AI training in a slightly backwards way. We've been obsessing over model size when the more important variable might be the quality of the feedback.
Alex: That's the implication. Data quality and feedback precision appear to matter at least as much as the number of parameters. A smaller model with better supervision can outperform a larger one trained on cruder signals.
Sam: It's a practical shift — and a fairly significant one for anyone building these systems.
Alex: It is. And it raises a broader question worth sitting with: in AI training, as in teaching, how much are we leaving on the table simply because we're not paying attention to the right things at the right moments? Thanks for listening to ResearchPod.