Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluations of coding agents largely emphasize aggregate task success, providing limited insight into why agents fail when repairing scientific software. We introduce \textbf{SWE-bench Science}, a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. Each task is organized into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. Even the best-performing agent, \textbf{Claude Code with Opus-5 (max), achieves a pass@1 below 50\%}, highlighting the substantial challenges posed by scientific software engineering. We identify four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in our analysis. We further conduct a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context. The results show that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success. Together, SWE-bench Science provides a broad testbed for studying both the capabilities and failure mechanisms of coding agents in scientific software engineering.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at how AI coding assistants handle scientific software. So this paper is basically asking: can AI actually fix complex research code?
Alex: That's the core question. The study examines whether automated coding agents — the kind of AI tools that can read, edit, and debug software on their own — can successfully repair code used in real scientific research.
Sam: And the stakes are pretty high, right? If there's an undetected bug in research code, it can quietly corrupt the actual findings of a published study.
Alex: That is exactly the concern. Think about it this way: scientific software isn't just a convenience tool. It's part of the instrument. If a telescope has a flaw in its lens, every observation it makes is compromised. A bug in a physics simulation is the same kind of problem — the results look fine on the surface, but the underlying conclusions are wrong.
Sam: So how do you even test whether an AI can fix that kind of problem?
Alex: The researchers built a benchmark — essentially a standardized test — called SWE-bench Science. It covers one hundred and nineteen real repair tasks drawn from twenty different scientific fields. And they were careful to prevent the AI from cheating. They used what they call a Chain-of-Evidence Protocol, which separates the public debugging materials the AI can see from a hidden set of private tests it cannot access. So the AI can't just learn to pass the visible checks while ignoring the deeper problem.
Sam: That's a bit like a student who memorizes the practice exam answers but hasn't actually understood the subject.
Alex: Exactly that. And when they ran the top frontier models through this benchmark, the results were sobering. Even the most capable systems succeeded on fewer than half the tasks.
Sam: Less than half. Why are they struggling so much with scientific code specifically?
Alex: The authors identify four recurring failure patterns. The first is what they call a scientific abstraction deficit — the AI doesn't understand the underlying physical or mathematical rules the code is supposed to follow. The second is superficial exploration, where the agent looks at the most obvious parts of the codebase and misses the deeper structural issue. Third is incomplete system integration — fixing one part of the code without realizing it breaks something else downstream. And fourth is a failure to generalize scientific knowledge across different contexts.
Sam: So an agent might generate a patch that makes the visible test pass, but the private verification catches that it violated some fundamental constraint — like a physical law.
Alex: Precisely. Imagine a simulation of a moving object where the code is supposed to conserve energy. An AI might patch the code so the output numbers look reasonable, but if you check carefully, the total energy in the system is slowly drifting upward — which is physically impossible. The patch passed the surface test and broke the science.
Sam: Did the researchers look at whether giving the models extra scientific documentation — textbooks, domain explanations — actually helps them avoid these failures?
Alex: They did. They ran a paired comparison, giving some agents explicit scientific guidance and withholding it from others. And the finding is counterintuitive: scientific knowledge is not uniformly beneficial.
Sam: How so? Wouldn't more information always help?
Alex: When the guidance is accurate and well-targeted, it can help the agent focus and work more efficiently. But when the documentation is poorly matched to the actual problem — or contains a misconception — it can anchor the agent onto a flawed path. The AI latches onto the wrong framing and stops exploring alternatives.
Sam: So stripping away extraneous documentation sometimes produces a cleaner environment for the repair.
Alex: The evidence suggests that removing poorly aligned details reduces distraction more than it removes useful signal. It's a meaningful caution for anyone building these systems: more context is not automatically better context.
Sam: That has broader implications, doesn't it? If we're moving toward AI systems that do autonomous scientific work, they'll need to do more than pass a unit test. They'll need to understand what the code is actually supposed to represent in the real world.
Alex: That's where the research points. The authors suggest future systems will likely need to cross-validate their patches directly against physical laws and domain constraints — not just against the test suite. The benchmark is designed to make that gap visible, so the field knows what it's actually working toward.
Sam: It's a useful reminder that in research, the software and the science aren't separate things. A flaw in one is a flaw in the other.
Alex: And that's what makes this evaluation meaningful. It's not just asking whether AI can write code — it's asking whether AI can reason about what the code is supposed to mean. Those are very different questions, and right now, the gap between them is significant.
Sam: Thanks for walking through the mechanics of this.
Alex: Thank you for exploring it. That's our look at evaluating AI coding agents on scientific software — and what it will take to close the gap between passing a test and understanding the science behind it. Thanks for listening to ResearchPod.