Thibaud Gloaguen, Niels Mündler, Mark Müller, Veselin Raychev, Martin Vechev
4 min
Abstract
A widespread practice in software development is to tailor coding agents to repositories using context files, such as AGENTS.md, by either manually or automatically generating them. Although this practice is strongly encouraged by agent developers, there is currently no rigorous investigation into whether such context files are actually effective for real-world tasks. In this work, we study this question and evaluate coding agents' task completion performance in two complementary settings: established SWE-bench tasks from popular repositories, with LLM-generated context files following agent-developer recommendations, and a novel collection of issues from repositories containing developer-committed context files. Across multiple coding agents and LLMs, we find that context files tend to reduce task success rates compared to providing no repository context, while also increasing inference cost by over 20%. Behaviorally, both LLM-generated and developer-provided context files encourage broader exploration (e.g., more thorough testing and file traversal), and coding agents tend to respect their instructions. Ultimately, we conclude that unnecessary requirements from context files make tasks harder, and human-written context files should describe only minimal requirements.
Sam: Developer files edged up success by 4 percent over no file. AI ones dipped 3 percent lower. Both added steps—developers' about 3 extra—and costs.
Alex: Even hunting for the files themselves seems wasteful. What about project overviews, like folder maps—do they speed up finding spots?
Sam: No. Agents without files reached changed files quicker. Files didn't shorten that; some wasted steps re-reading instructions.
Alex: Maps distract more than help. What if projects lack other docs, like read-me notes?
Sam: They stripped docs, leaving only context files. AI-generated ones then beat developer ones by 3 percent. But with existing docs, files duplicated info, adding noise.
Alex: Files fill small gaps at best. What do logs say about how files shape agent actions?
Sam: Agents follow closely—if a file mentions a test command, they use it 1.6 times per task versus almost never without. This means extra searches, reads, writes, and tests, spreading effort thin.
Alex: More rules add mental overhead. Like juggling extra guidelines before acting?
Sam: Yes. Reasoning tokens—internal planning notes—rose 14 to 22 percent with files, making decisions harder.
Alex: No magic recipe yet. Where does this leave real projects?
Sam: Files have small effects, raise costs over 20 percent. Agents creating slim, task-focused files on the fly could help—early signs show boosts.
Alex: But limits? Like only Python?
Sam: Yes, mainly Python, where AIs know common tools. Future work could check rarer setups, code quality, or agents learning better files over time.
Alex: A snapshot questioning a practice in over 60,000 repositories. Developers might rethink routine use. Thanks, Sam—that's a clear synthesis. Thanks for joining me on ResearchPod.