Author-updated Summary
Verified author edit
Modern coding agents are often evaluated by their ability to generate a correct patch, but this end-to-end success masks a critical upstream failure point: the agent's ability to identify and retrieve the necessary files from a repository. This paper introduces Agent Retrieval Bench, a benchmark designed to isolate this context-acquisition layer. It asks: can a retrieval system identify the specific files an agent needs to read next, given a realistic coding workflow signal, while operating under strict constraints and avoiding 'shortcuts' like direct semantic similarity?
The authors constructed a dataset of 427 samples across 25 repositories, categorized into four positive-retrieval tasks (code2test, comment2context, trace2code, and edit2ripple) and a selective retrieval subset. Unlike traditional code search, which relies on semantic similarity between a query and a file, this benchmark defines relevance based on the next step in a developer's workflow. The evaluation uses frozen base-commit snapshots to prevent data leakage and includes rigorous controls to ensure that retrieval is based on genuine task requirements rather than trivial matches. The authors evaluate various methods, including lexical search, RepoMap, and open-source embeddings, using metrics like Recall@k, Mean Reciprocal Rank (MRR), and Budgeted Context Yield (BCY).
The study finds that no single retrieval family dominates across all tasks, suggesting that structural and semantic methods are complementary. While some models perform well on specific metrics (e.g., Qwen3-Embedding-4B for MRR, RepoMap for context yield), performance varies significantly by task. A key finding is the 'calibration gap': models struggle to correctly abstain from returning files when no relevant context exists, even when tested against counterfactual controls. Furthermore, analysis of logged agent trajectories shows that even in successful workflows, agents often fail to touch gold-standard files in 27-35% of cases, highlighting a persistent bottleneck in current agentic systems.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a paper that explores how AI coding agents find the information they need to do their jobs.
Sam: Thanks, Alex. The central puzzle is this: we usually judge AI coding agents by whether they successfully fix a bug. But the research suggests they often fail long before they type a single line of code — because they can't find the right files in a massive codebase to begin with.
Alex: So the paper is asking: why do these agents struggle to find the right evidence, even when the answer is sitting right there in the repository?
Sam: Exactly. Imagine you're a student trying to answer an essay question, but instead of searching for sources that actually address the question, you just look for articles that share the same words as the question. You'd end up with a pile of loosely related material and miss the one source that actually matters. That's essentially what these agents do. The researchers call what they're actually looking for "agentic relevance" — not just files that sound similar to the task, but files that are the genuinely useful next step in the workflow.
Alex: So it's less about vocabulary matching and more about understanding the structure of the problem.
Sam: Right. And to study this properly, the researchers built a benchmark — think of it as a standardized test — called Agent Retrieval Bench. It isolates just the information-gathering step, separating it from the code-writing step, so you can measure each one independently.
Alex: Why separate them? If an agent fails to fix a bug, isn't it usually because it just isn't smart enough to write the code?
Sam: That's a common assumption, but the data challenges it. By isolating the retrieval step, they found that even when agents are given the freedom to explore a codebase interactively, they completely miss the necessary files on more than a quarter of all tasks. The failure isn't always in the reasoning — it's in the initial search.
Alex: So the bottleneck isn't just the AI's intelligence. It's how the AI navigates the codebase in the first place.
Sam: Precisely. If an agent doesn't pull the right context into its working memory early on, it's essentially flying blind for everything that follows. To make sure the test is fair, the researchers use what they call "base-commit hygiene." They freeze the codebase at the exact moment before any fix was applied — like preserving a crime scene — and strip out any hints about which files were eventually changed. That forces the agent to actually search for the answer rather than accidentally reading it off a cheat sheet.
By decoupling context acquisition from patch generation, this benchmark provides a standardized way to diagnose why coding agents fail. It establishes that retrieval quality is a distinct, measurable, and critical component of agent performance. The findings suggest that improving the 'upstream' retrieval process—specifically by better aligning it with workflow-dependent relevance—is essential for building more reliable and efficient coding agents.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Alex: So they're testing whether the agent can find the needle in the haystack without being told where the needle is.
Sam: Exactly. And once the environment is frozen, they measure performance in a couple of ways. The simpler one is recall — did the right file end up anywhere in the agent's list of results? But they also use a metric called Budgeted Context Yield, or BCY. This one captures something subtler: an agent has a limited amount of working memory, so even if it finds the right file, if that file is buried under hundreds of irrelevant ones, the agent still can't use it effectively.
Alex: So it's not just about finding the needle — it's about how much of the needle you can actually hold in your hand at once.
Sam: That's a good way to put it. And when you measure things that carefully, the results get interesting. No single search method comes out on top across all tasks. Some tools work by reading the structural map of a codebase — understanding how files connect to each other, like a diagram of a city's road network. Others use more mathematical approaches to match the meaning of a query to the content of a file. Each has strengths, but neither dominates.
Alex: So what determines which one works better?
Sam: The type of task matters a lot. Finding a test file to verify a bug fix calls for a different strategy than making sense of a code review comment. If you collapse everything into one average score, you hide those differences entirely. The paper argues that's been a significant blind spot in how we evaluate these systems.
Alex: So a one-size-fits-all search engine isn't the answer.
Sam: Not even close. The study's position is that agentic retrieval isn't one single relationship between a query and a file. It's a collection of different signals — things like the history of what the agent has already looked at, or the structural links between files — and those signals need to be weighted differently depending on what the agent is trying to accomplish.
Alex: There's also something the paper raises about what happens when the agent can't find anything useful at all, right?
Sam: Yes. Ideally, an agent that can't locate a relevant file should be able to say so — to abstain rather than guess. But the paper finds that current methods aren't reliable at this yet. Agents tend to return something even when nothing useful exists, which can be worse than returning nothing at all, because the downstream code-writing step then works from bad information.
Alex: So the agent isn't just failing to think — it's failing to see. And when it pretends it can see, that makes things worse.
Sam: That's a fair summary. The core argument of the paper is that we've been measuring the wrong thing. We focus on whether the agent produces a correct fix at the end, but we don't measure whether it found the right information to work from in the first place. Agent Retrieval Bench is an attempt to make that invisible step visible and measurable.
Alex: And once you can measure it, you can actually improve it.
Sam: That's the hope. Right now, even the best models struggle with this. But having a clear, standardized way to test retrieval performance separately from code generation gives researchers a much more precise target to aim at. The paper's position is that until we treat context acquisition as its own distinct challenge, we'll keep hitting the same ceiling — agents that are capable of writing good code, but can't find their way to the right starting point.
Alex: So the lesson is that intelligence without navigation is only half the picture. Thanks for walking us through this, Sam — and thanks to everyone listening to ResearchPod.