Yusheng Zhao, Bowei He, Xue Liu, Mingxiao Li
8 min
Abstract
Large language model agents are increasingly used for scientific discovery. However, existing benchmarks mix law inference with programming, score data fitting instead of prediction on unseen inputs, and do not separate empirical evidence from pretrained priors. We introduce SciGame, a controlled benchmark that converts scientific literature into structured scientific cards. Each card yields a pair of Original and dimensionally consistent Counterfactual environments with the same task interface. SciGame contains a Historical collection of canonical laws and a live Temporal collection curated from papers published after model knowledge cutoffs. It evaluates agents in candidate-guided Identification and open-ended Discovery settings. Each setting runs under Prior-only, Fixed, and Adaptive protocols. Experiments on five frontier models yield three findings. First, familiar priors hurt performance once the governing relation is altered. Second, adaptive experimentation helps the strongest models but harms weaker ones under limited budgets. Third, good explanations can hide quantitative failures. Recovering the exact symbolic form remains the main bottleneck, even when agents interpret evidence correctly. Current agents can investigate systems actively, but inducing exact laws in unfamiliar settings remains an open challenge.
Alex: That's the intended test. The authors also check that each world's true formula fails prediction testing in its counterpart. So the paired worlds are empirically distinguishable. But those mutations are benchmark constructions, not newly discovered physical laws.
Sam: If you provide possible formulas, distinguishing them seems easier than inventing a relationship. Do they separate those tasks?
Alex: Yes. In Identification, the agent receives a formula combining the original and altered expressions through a hidden parameter. In Discovery, it gets no candidate expressions. Both require a mathematical predictor, an explanation, and applicability conditions. Predictors use structured arithmetic expressions, not arbitrary programs.
Sam: That removes some software-engineering demands. How do they distinguish what the model already knows from what it learns during the task?
Alex: They compare three observation protocols. Prior-only gives no measurements. Fixed supplies a preset batch. Adaptive lets the agent propose experiments to an Oracle—the simulator returning measurements or constraint feedback. During exploration, the agent must state competing hypotheses and predict their results before seeing the observations.
Sam: Choosing when to stop could affect the result as much as choosing where to measure. Is stopping part of the test?
Alex: It is. The agent reports confidence in its hypotheses and must submit once the confidence rule requires it. Breaking that rule makes the episode unsuccessful, even if its predictor remains numerically assessable. Reported confidence is used for protocol checks; the authors don't assume it represents calibrated probabilities.
Sam: Before we get to the results, how broad is the test? Are these all familiar textbook relationships?
Alex: There are 44 task families, mostly from physics. A historical collection contains established relationships. A temporal collection draws from recent papers, with date filters tied to reported model knowledge cutoffs. That pipeline can be rerun as the cutoffs move.
Sam: Dates help with memorization, but they don't prove a model has never encountered the material. And fresh papers still need translating into these cards.
Alex: Both are limitations the authors acknowledge. A language model extracts the relationships, and a separate model reviews the cards. The date filter reduces contamination but cannot rule it out. The temporal environments also simplify selected reported results; they don't reproduce entire papers.
Sam: Then what counts as success? Your opening distinguished a convincing explanation from getting the law right.
Alex: The benchmark keeps those separate. A prediction score measures error on hidden inputs. Weak episode success requires predictions within tolerance and protocol compliance. Strict success additionally requires recovering the exact reference expression. A language-model judge separately assesses the explanation's scientific content, support, and scope.
Sam: That strict requirement could reject a useful approximation. Does the difference between numerical prediction and exact recovery actually change the headline?
Alex: Substantially. In adaptive discovery of historical counterfactuals, GPT-6 Astra achieves 95 percent weak success. Its strict success is under two percent. So it often predicts accurately within the evaluation tolerances without recovering the exact expression.
Sam: That's a narrower failure than simply misunderstanding the system. How does active experimentation compare with receiving observations?
Alex: For GPT-6 Astra, strict success on temporal counterfactual discovery rises from about ten percent with preset observations to about seventeen percent adaptively. The fixed condition supplies four observations. Several weaker models show no improvement or decline when choosing experiments themselves.
Sam: But adaptive agents can collect more evidence. Does that show better experiment selection, or just a benefit from longer investigations?
Alex: The budget comparisons qualify the headline. With small matched observation budgets, preset observations work better for counterfactual discovery. Adaptive discovery starts slowly, while supplied candidate formulas can often be distinguished immediately. Successful open-ended searches can require much longer interactions. The overall adaptive advantage isn't evidence of greater efficiency.
Sam: So active investigation is neither automatically better nor automatically cheaper. And an explanation still can't tell us whether the final predictor works.
Alex: The study finds cases with strong explanation scores and poor numerical performance. Its failure analysis also distinguishes incorrect relationships from failure to recover the exact expression. I'd read that as a reason to inspect all the metrics, not treat either fluent explanation or strict symbolic success as the whole story.
Sam: What most limits applying these results to scientific agents outside this benchmark?
Alex: The environments are deterministic, without stochastic measurement noise. They use a limited set of synthetic mutations, and the collections use different mutation schemes. Each condition has only a few repetitions, so small differences are uncertain. And explanation judging relies on a model that is itself being evaluated.
Sam: Given those boundaries, who should read the full document, and where should they start?
Alex: Researchers building scientific agents or evaluating automated discovery should read it. Start with Experimental Framework and Protocol, then Evaluation Metrics. After that, inspect the weak-versus-strict success comparison in Additional Experimental Results. It reveals what the headline failures mean. For everyone else: a convincing explanation, a useful predictor, and a recovered law are three different achievements.
Sam: Keep those distinctions in mind when the next agent reports a discovery.