Zhipeng Xu, Jiahao Lu, Yining Zheng, Yuxin Wang, Xipeng Qiu
5 min
Abstract
Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluations of coding agents largely emphasize aggregate task success, providing limited insight into why agents fail when repairing scientific software. We introduce \textbf{SWE-bench Science}, a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. Each task is organized into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. Even the best-performing agent, \textbf{Claude Code with Opus-5 (max), achieves a pass@1 below 50\%}, highlighting the substantial challenges posed by scientific software engineering. We identify four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in our analysis. We further conduct a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context. The results show that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success. Together, SWE-bench Science provides a broad testbed for studying both the capabilities and failure mechanisms of coding agents in scientific software engineering.
Alex: Precisely. Imagine a simulation of a moving object where the code is supposed to conserve energy. An AI might patch the code so the output numbers look reasonable, but if you check carefully, the total energy in the system is slowly drifting upward — which is physically impossible. The patch passed the surface test and broke the science.
Sam: Did the researchers look at whether giving the models extra scientific documentation — textbooks, domain explanations — actually helps them avoid these failures?
Alex: They did. They ran a paired comparison, giving some agents explicit scientific guidance and withholding it from others. And the finding is counterintuitive: scientific knowledge is not uniformly beneficial.
Sam: How so? Wouldn't more information always help?
Alex: When the guidance is accurate and well-targeted, it can help the agent focus and work more efficiently. But when the documentation is poorly matched to the actual problem — or contains a misconception — it can anchor the agent onto a flawed path. The AI latches onto the wrong framing and stops exploring alternatives.
Sam: So stripping away extraneous documentation sometimes produces a cleaner environment for the repair.
Alex: The evidence suggests that removing poorly aligned details reduces distraction more than it removes useful signal. It's a meaningful caution for anyone building these systems: more context is not automatically better context.
Sam: That has broader implications, doesn't it? If we're moving toward AI systems that do autonomous scientific work, they'll need to do more than pass a unit test. They'll need to understand what the code is actually supposed to represent in the real world.
Alex: That's where the research points. The authors suggest future systems will likely need to cross-validate their patches directly against physical laws and domain constraints — not just against the test suite. The benchmark is designed to make that gap visible, so the field knows what it's actually working toward.
Sam: It's a useful reminder that in research, the software and the science aren't separate things. A flaw in one is a flaw in the other.
Alex: And that's what makes this evaluation meaningful. It's not just asking whether AI can write code — it's asking whether AI can reason about what the code is supposed to mean. Those are very different questions, and right now, the gap between them is significant.
Sam: Thanks for walking through the mechanics of this.
Alex: Thank you for exploring it. That's our look at evaluating AI coding agents on scientific software — and what it will take to close the gap between passing a test and understanding the science behind it. Thanks for listening to ResearchPod.