Guowei Xu, Jiaru Zou, Mete Erdogan, Haotian Ye, Pan Lu, James Zou
8 min
Abstract
Scientific discovery requires ideas that are original, useful, and non-obvious. Yet AI scientists are typically given as much task information as possible during idea generation, which can anchor generation toward familiar approaches. Inspired by the cognitive framework of Blind Variation and Selective Retention (BVSR), we propose a discovery framework that separates these two roles of information: it generates proposals under deliberately limited task information to broaden exploration, then learns from proposals previously verified on the same task and their outcomes to retain promising candidates for costly empirical evaluation. We evaluate BVSR across five proposer models and 12 scientific tasks. Limiting task information substantially increases proposal originality and non-obviousness, and produces a larger share of highly creative and high-utility proposals than existing state-of-the-art prompting-based baselines. Learning from prior proposal outcomes on the task further improves candidate selection, discarding half of the candidates while retaining most top-performing proposals. When incorporated into existing AI-scientist workflows, BVSR achieves stronger discoveries across gene expression analysis, drug response prediction, anomaly detection, scaling law discovery, and brain activity prediction. These results suggest that effective AI scientists should use information differently across stages of discovery: constrain task-specific information during exploration while exploiting accumulated empirical experience on the task.
Sam: And did that produce anything more concrete than an unusual analogy?
Alex: One successful gene-ranking program borrowed the nudged elastic band method, which finds a minimum-energy path in transition-state theory. The program follows how a gene's expression pattern smooths toward uniformity across the tissue. It scores a force component perpendicular to that path, which the authors connect to spatial pattern strength.
Sam: So the borrowed mechanism becomes part of an executable scoring method. How did it fare against the existing alternatives?
Alex: On that benchmark, it beat both the original AI-scientist workflow and SPARK-X, the strongest existing baseline reported for the task. That's one illustrative discovery. The broader question is whether restricting information helped systematically, rather than producing one lucky transfer.
Sam: How broad was that test, and how did they keep creativity from meaning whatever looked interesting afterward?
Alex: They tested five proposer models across twelve scientific tasks. These covered biology, machine learning, spacecraft trajectory design, and brain activity forecasting. For each model, task, and information level, they generated 112 proposals.
Sam: Repeated generation gives them a way to measure diversity. But diversity alone would reward a collection of unusual failures.
Alex: Their creativity measure combines originality, tested utility, and non-obviousness: whether success could have been anticipated from existing knowledge. An idea needs to work to score well. Utility comes from executing its code under the task's evaluator.
Sam: Originality is the slippery part. Does it mean an idea is new to science, or just uncommon among this model's suggestions?
Alex: The latter. They group proposals expressing the same mechanism and estimate originality from how rarely each appears. That is generative rarity, not novelty relative to the scientific literature. A rarely proposed mechanism can still come from an established paper.
Sam: That distinction changes how I'd interpret “creative discovery.” What happened when they varied the information rather than just the instructions?
Alex: More task information generally reduced the share of proposals among the benchmark's highest-ranked for creativity or utility. Average originality and non-obviousness also declined. Limited-information generation led the prompting baselines on their reported measures, including top utility.
Sam: Could the gain come from a better-written prompt rather than the missing information?
Alex: They included the same blind-variation prompt with the full task description as a control. Its advantage shrank when given the complete description. That supports information restriction as a source of the gains, although the findings remain tied to these models and tasks.
Sam: Now for the other half. Wider exploration could mean more bad proposals to test. How do they avoid spending the savings on a larger pile of failures?
Alex: Selective retention is their learned screening stage. It uses previous proposals and measured outcomes from the same task. The model learns to keep strong candidates above the discard boundary, rather than predict every candidate's exact performance.
Sam: That objective sounds consequential. Getting the middle of the ranking wrong may not matter if the exceptional ideas survive.
Alex: Their training compares high-utility proposals against proposals from the remainder. It rewards scoring the strong ones higher. They trained on roughly forty-one thousand previously tested proposals, separate from those used to evaluate screening.
Sam: How much survived the screening, compared with simply throwing candidates away at random?
Alex: When the selector discarded half the candidates, it retained eighty-two percent of proposals in the top five percent for utility. Random discarding would retain about half. So it preserved most exceptional performers, but still lost some.
Sam: That sounds useful, but not free of risk. Did simpler judges, or a model trained to predict performance, do as well?
Alex: No. The learned selector preserved top candidates better than prompting-based judges, judges shown similar verified examples, and supervised performance regression. Then the authors added both stages to SimpleTES, an existing discovery workflow. Across the tested applications, it achieved higher final scores and surpassed the original workflow's final score with fewer verified proposals.
Sam: Where should we draw the boundary around that result? Would I trust the selector on a new scientific problem?
Alex: That transfer isn't established here. Training and evaluation use different proposals, but the same tasks. The selector benefits from accumulated outcomes on those tasks; this isn't evidence that it arrives ready to screen an unfamiliar problem. And the empirical tests are computational, not wet-lab validations.
Sam: There's also judgment inside the creativity score. How much confidence should we place in that?
Alex: Language models judge mechanism equivalence and non-obviousness. The authors checked both against doctoral-level researchers and found meaningful, imperfect agreement. Human evaluators also rated blind-variation proposals as less obvious than analogical-reasoning proposals. That supports the measure without making it objective ground truth.
Sam: Given those boundaries, who should read the full document, and where should they start?
Alex: Researchers building automated discovery workflows should read the baseline comparison, especially limited versus full information under the same prompt. Then read Learning to Select Top Proposals for the screening objective and its training requirements. The Applications section shows whether those components improve an actual search workflow.
Sam: And what should everyone else remember when deciding how much context to give a model?
Alex: In these experiments, exploration and selection benefited from different information. Give ideas room to wander; use tested experience to decide which deserve investment.
Sam: Keep that distinction in mind.