A widespread practice in software development is to tailor coding agents to repositories using context files, such as AGENTS.md, by either manually or automatically generating them. Although this practice is strongly encouraged by agent developers, there is currently no rigorous investigation into whether such context files are actually effective for real-world tasks. In this work, we study this question and evaluate coding agents' task completion performance in two complementary settings: established SWE-bench tasks from popular repositories, with LLM-generated context files following agent-developer recommendations, and a novel collection of issues from repositories containing developer-committed context files. Across multiple coding agents and LLMs, we find that context files tend to reduce task success rates compared to providing no repository context, while also increasing inference cost by over 20%. Behaviorally, both LLM-generated and developer-provided context files encourage broader exploration (e.g., more thorough testing and file traversal), and coding agents tend to respect their instructions. Ultimately, we conclude that unnecessary requirements from context files make tasks harder, and human-written context files should describe only minimal requirements.
Alex: Welcome to another episode of ResearchPod.
Sam: Today, we're looking at something notable in software development. Developers are adding special instruction files to their code projects to help AI tools fix bugs.
Alex: These AI tools are called coding agents, right? Like a smart assistant that spots a bug, reads files, edits code, and runs tests—all on its own?
Sam: Exactly. Developers now include files like AGENTS.md to guide them better. Over 60,000 public GitHub repositories have them, following advice from AI companies. These files overview the codebase—like a workshop map—and list tools, such as test commands.
Alex: So a developer adds one to help the AI navigate a tricky project faster. Does it actually work that way?
Sam: A recent study tested this on real GitHub tasks, like bug fixes. They compared agents with no file, AI-generated files, and developer-written ones. The paper suggests context files often hurt performance. Success rates dropped versus no file, while computing time rose by over 20 percent.
Alex: Huh. Agents solve fewer problems and take more effort? Why would extra instructions backfire?
Sam: Traces of agent actions show they explore more—extra tests, more files checked—which follows instructions but adds steps without fixes. Developer files lifted success by about 4 percent on average. AI-generated ones lowered it by 3 percent. This held across models.
Alex: They follow advice to explore, but it doesn't pay off. How did researchers ensure a fair test?
Sam: They picked Python projects with context files, built-in tests, and at least 400 past fixes. From thousands of pull requests—proposed changes linked to issues—they selected 138 reliable ones from 12 projects. Pull requests suggest code updates to solve issues.
Alex: And they made changes testable, even in small projects.
Sam: Yes. An AI created setup scripts and clear task instructions from fix details, without solution hints—like what should happen and steps to verify. For verification, an AI generated new tests that fail on broken code but pass the real fix. They combined these with repo tests, covering about three-quarters of changed code.
Alex: Like rebuilding a referee's rulebook from game highlights to judge players fairly. That let them compare agents head-to-head.
Sam: Yes. Agents with files solved fewer tasks, used more steps, and had over 20 percent higher costs. Logs showed more broad searches and early tests, delaying focus on key files—like checking every room before the leaky pipe.
Alex: Extra wandering at first. Do developer files work better than AI ones?
Sam: Developer files edged up success by 4 percent over no file. AI ones dipped 3 percent lower. Both added steps—developers' about 3 extra—and costs.
Alex: Even hunting for the files themselves seems wasteful. What about project overviews, like folder maps—do they speed up finding spots?
Sam: No. Agents without files reached changed files quicker. Files didn't shorten that; some wasted steps re-reading instructions.
Alex: Maps distract more than help. What if projects lack other docs, like read-me notes?
Sam: They stripped docs, leaving only context files. AI-generated ones then beat developer ones by 3 percent. But with existing docs, files duplicated info, adding noise.
Alex: Files fill small gaps at best. What do logs say about how files shape agent actions?
Sam: Agents follow closely—if a file mentions a test command, they use it 1.6 times per task versus almost never without. This means extra searches, reads, writes, and tests, spreading effort thin.
Alex: More rules add mental overhead. Like juggling extra guidelines before acting?
Sam: Yes. Reasoning tokens—internal planning notes—rose 14 to 22 percent with files, making decisions harder.
Alex: No magic recipe yet. Where does this leave real projects?
Sam: Files have small effects, raise costs over 20 percent. Agents creating slim, task-focused files on the fly could help—early signs show boosts.
Alex: But limits? Like only Python?
Sam: Yes, mainly Python, where AIs know common tools. Future work could check rarer setups, code quality, or agents learning better files over time.
Alex: A snapshot questioning a practice in over 60,000 repositories. Developers might rethink routine use. Thanks, Sam—that's a clear synthesis. Thanks for joining me on ResearchPod.