Bowen Qin, Yi Xie
6 min
Modern coding agents are usually evaluated by whether they eventually produce a correct patch, but patch generation depends on an earlier context-acquisition stage: finding the repository files needed for the task. We introduce Agent Retrieval Bench, a file-level benchmark for this upstream retrieval problem. Samples are built from real coding-workflow signals and evaluated against frozen base-commit repositories, with relevance defined by what an agent needs next rather than direct query-file semantic similarity. The benchmark covers four positive-retrieval tasks: code2test, comment2context, trace2code, and edit2ripple; a fifth subset evaluates selective retrieval using natural evidence-backed no-gold cases and counterfactual wrong-repository controls. Agent Retrieval Bench contains 427 samples across 25 repositories: 345 positive examples, 50 natural no-gold examples, and 32 counterfactual controls. The corpus includes 308 base-commit snapshots, 392,000 files, and 7.9 million chunks. We evaluate lexical retrieval, RepoMap, open-source embeddings, selective abstention, and logged agent context selection. No single retrieval family dominates: Qwen3-Embedding-4B has the best sample-weighted MRR on positive samples, Qwen3-Embedding-8B the best Recall@20, and RepoMap the best budgeted context yield at 8K tokens, with task-level winners differing substantially. Selective thresholds calibrated with counterfactual controls do not improve selective success on natural no-gold cases, revealing a calibration gap. Logged trajectories also miss every gold file on 27-35 percent of samples. A controlled seed-intervention pilot finds that retrieval-derived initial context yields higher file F1 with less post-seed exploration than random non-gold context, while oracle gold context shows substantial remaining headroom.
Modern coding agents are often evaluated by their ability to generate a correct patch, but this end-to-end success masks a critical upstream failure point: the agent's ability to identify and retrieve the necessary files from a repository. This paper introduces Agent Retrieval Bench, a benchmark designed to isolate this context-acquisition layer. It asks: can a retrieval system identify the specific files an agent needs to read next, given a realistic coding workflow signal, while operating under strict constraints and avoiding 'shortcuts' like direct semantic similarity?
The authors constructed a dataset of 427 samples across 25 repositories, categorized into four positive-retrieval tasks (code2test, comment2context, trace2code, and edit2ripple) and a selective retrieval subset. Unlike traditional code search, which relies on semantic similarity between a query and a file, this benchmark defines relevance based on the next step in a developer's workflow. The evaluation uses frozen base-commit snapshots to prevent data leakage and includes rigorous controls to ensure that retrieval is based on genuine task requirements rather than trivial matches. The authors evaluate various methods, including lexical search, RepoMap, and open-source embeddings, using metrics like Recall@k, Mean Reciprocal Rank (MRR), and Budgeted Context Yield (BCY).
The study finds that no single retrieval family dominates across all tasks, suggesting that structural and semantic methods are complementary. While some models perform well on specific metrics (e.g., Qwen3-Embedding-4B for MRR, RepoMap for context yield), performance varies significantly by task. A key finding is the 'calibration gap': models struggle to correctly abstain from returning files when no relevant context exists, even when tested against counterfactual controls. Furthermore, analysis of logged agent trajectories shows that even in successful workflows, agents often fail to touch gold-standard files in 27-35% of cases, highlighting a persistent bottleneck in current agentic systems.
By decoupling context acquisition from patch generation, this benchmark provides a standardized way to diagnose why coding agents fail. It establishes that retrieval quality is a distinct, measurable, and critical component of agent performance. The findings suggest that improving the 'upstream' retrieval process—specifically by better aligning it with workflow-dependent relevance—is essential for building more reliable and efficient coding agents.
Sam: Exactly. And once the environment is frozen, they measure performance in a couple of ways. The simpler one is recall — did the right file end up anywhere in the agent's list of results? But they also use a metric called Budgeted Context Yield, or BCY. This one captures something subtler: an agent has a limited amount of working memory, so even if it finds the right file, if that file is buried under hundreds of irrelevant ones, the agent still can't use it effectively.
Alex: So it's not just about finding the needle — it's about how much of the needle you can actually hold in your hand at once.
Sam: That's a good way to put it. And when you measure things that carefully, the results get interesting. No single search method comes out on top across all tasks. Some tools work by reading the structural map of a codebase — understanding how files connect to each other, like a diagram of a city's road network. Others use more mathematical approaches to match the meaning of a query to the content of a file. Each has strengths, but neither dominates.
Alex: So what determines which one works better?
Sam: The type of task matters a lot. Finding a test file to verify a bug fix calls for a different strategy than making sense of a code review comment. If you collapse everything into one average score, you hide those differences entirely. The paper argues that's been a significant blind spot in how we evaluate these systems.
Alex: So a one-size-fits-all search engine isn't the answer.
Sam: Not even close. The study's position is that agentic retrieval isn't one single relationship between a query and a file. It's a collection of different signals — things like the history of what the agent has already looked at, or the structural links between files — and those signals need to be weighted differently depending on what the agent is trying to accomplish.
Alex: There's also something the paper raises about what happens when the agent can't find anything useful at all, right?
Sam: Yes. Ideally, an agent that can't locate a relevant file should be able to say so — to abstain rather than guess. But the paper finds that current methods aren't reliable at this yet. Agents tend to return something even when nothing useful exists, which can be worse than returning nothing at all, because the downstream code-writing step then works from bad information.
Alex: So the agent isn't just failing to think — it's failing to see. And when it pretends it can see, that makes things worse.
Sam: That's a fair summary. The core argument of the paper is that we've been measuring the wrong thing. We focus on whether the agent produces a correct fix at the end, but we don't measure whether it found the right information to work from in the first place. Agent Retrieval Bench is an attempt to make that invisible step visible and measurable.
Alex: And once you can measure it, you can actually improve it.
Sam: That's the hope. Right now, even the best models struggle with this. But having a clear, standardized way to test retrieval performance separately from code generation gives researchers a much more precise target to aim at. The paper's position is that until we treat context acquisition as its own distinct challenge, we'll keep hitting the same ceiling — agents that are capable of writing good code, but can't find their way to the right starting point.
Alex: So the lesson is that intelligence without navigation is only half the picture. Thanks for walking us through this, Sam — and thanks to everyone listening to ResearchPod.