NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers? | Yuru Wang et al. | ResearchPod

Transcript: NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

Alex: Welcome to another episode of ResearchPod. Today we're asking a question that sounds simple but turns out to be surprisingly difficult to answer: can AI systems actually make new scientific discoveries, or are they just very good at repeating what they've already seen?

Sam: That distinction matters a lot. There's a big difference between an AI that can look up a known solution and one that can genuinely figure something out from scratch.

Alex: And that's exactly what this paper is trying to test. The researchers built a new framework called NatureBench. The idea is to take real scientific problems from top research journals and see if AI agents can beat the best published results—without being shown how the original researchers did it.

Sam: So how do they stop the AI from just finding the answer in the original paper?

Alex: That's the key design challenge. They built an automated system called NatureGym. Think of it like a sealed exam room. The AI gets all the data and tools it needs to run an experiment, but it's cut off from the internet and from the original paper's source code. There's what the researchers call an "information firewall"—it strips out the original author's method entirely, so the AI has to come up with its own approach.

Sam: Like a blind taste test. If the AI can still match or beat the original result without seeing the recipe, that's genuine work.

Alex: That's the logic. And the results are sobering. They tested ten different AI agents across 90 of these scientific tasks. The strongest model only beat the original published results about 18% of the time.

Sam: Only 18%? That's quite low. What's going wrong the other 82% of the time?

Alex: The researchers found two main failure modes. Sometimes the agent picks the wrong method entirely—it misreads what kind of problem it's dealing with. Other times, it simply runs out of computing resources before it can finish. Scientific problems are often much more demanding than the tasks these systems are typically trained on.

Sam: And when they do succeed—what's actually happening there?

Alex: This is where it gets interesting. When the agents do beat the published results, they're usually not doing something genuinely new. The researchers identified a pattern they call "methodological translation." Imagine you're given a complex physics problem. A skilled AI doesn't necessarily solve the physics. Instead, it recognises that the structure of the problem resembles a standard mathematical puzzle it's seen many times before, and it solves that puzzle instead.

Sam: So it's not thinking about the science. It's finding a familiar shape underneath an unfamiliar problem and solving that instead.

NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

Yuru Wang, Lejun Cheng, Yuxin Zuo, Sihang Zeng, Bingxiang He, Che Jiang, Junlin Yang, Yuchong Wang, Kaikai Zhao, Weifeng Huang, Kai Tian, Zhenzhao Yuan, Jincheng Zhong, Weizhi Wang, Ning Ding, Bowen Zhou, Kaiyan Zhang

Jun 23, 2026·3:32·arxiv·Original Paper

NatureBenchNatureGymAI Coding AgentsMethodological TranslationEnvironment FragmentationLiquid Time-Constant (LTC) Networks

Listen

4 min

Briefing

Abstract

We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. Evaluating ten frontier agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 17.8% of tasks under the g>0.1 criterion. Analysis of method pathways reveals that agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding. We release the benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction. Code: https://github.com/FrontisAI/NatureBench

Research Question and Approach

NatureBench addresses a critical gap in evaluating AI agents: can they move beyond reproducing existing code to actually discovering new scientific methods? While prior benchmarks focus on code replication or engineering optimization, NatureBench evaluates agents on 90 tasks distilled from peer-reviewed Nature-family publications. The authors developed NatureGym, an automated pipeline that converts scientific papers into containerized, reproducible task environments. By enforcing an information firewall that removes the original source code and methods, the authors force agents to solve scientific problems from scratch using only the provided data and task descriptions.

Evaluation and Findings

The study evaluated ten frontier AI agent configurations, including models like Claude Opus 4.7 and GPT-5.5, under a strict web-search-disabled protocol. The results show that even the most capable agents rarely surpass the published state-of-the-art (SOTA). The strongest model achieved a SOTA-surpassing performance on only 17.8% of tasks. Analysis of the agents' successful pathways reveals a pattern of methodological translation: agents often reframe complex scientific problems into standard supervised prediction tasks they have encountered during training, rather than inventing novel scientific approaches. When agents failed, the primary drivers were incorrect method selection and insufficient compute budgets rather than a fundamental misunderstanding of the task requirements.

Why It Matters

This research highlights that while AI agents are becoming proficient at coding and standard machine learning tasks, they are not yet autonomous scientific researchers. The reliance on methodological translation suggests that current agents are limited by their training data distribution and struggle to perform the creative, domain-specific reasoning required for scientific discovery. By providing a standardized, reproducible benchmark, NatureBench offers a rigorous framework for tracking whether future AI models can bridge the gap between mere implementation and genuine scientific innovation.

Key Terms and Definitions

NatureGym — An automated pipeline that extracts and containerizes scientific tasks from research papers, ensuring reproducibility by standardizing environments.
Methodological Translation — A strategy where an agent converts a complex, domain-specific scientific problem into a standard supervised learning task that it already knows how to solve.
Information Firewall — A constraint applied to the benchmark where the original source code and method descriptions are removed, forcing the agent to discover a solution independently.
Surpass-SOTA (g > 0.1) — A primary performance metric defined as the relative gap by which an agent's solution exceeds the published state-of-the-art performance in a paper.
Environment Fragmentation — The lack of standardized, containerized setups in research benchmarks, which makes it difficult to independently verify or compare agent performance across different scientific studies.

NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?: Research Paper Podcast