Wenyu Du, Stephen Chung
9 min
Scientific discovery via AI has largely been restricted to domains with well-defined metrics or rigid pipelines. This paper investigates whether AI agents can autonomously perform open-ended scientific discovery—tasks lacking objective metrics—by simulating a decentralized ecosystem where agents must rely on subjective judgment and persistent exploration.
The authors utilize Station, an open-world multi-agent research environment, and extend it with two primary mechanisms:
To test this, the authors constructed three open-ended tasks based on ICLR oral papers, withholding the original findings. They compared Station against two centralized baselines: Codex Multiagent-v2 and AI Scientist-v2. Performance was measured by the agents' ability to rediscover specific findings partitioned into discrete criteria.
Station achieved significantly higher rediscovery rates, averaging 62.7% across the tasks compared to 14.4–20.6% for the baselines. Ablation studies confirmed that the combination of the Supervisor and Meta Reflection mechanisms was crucial for improving research continuity and coverage. Furthermore, the agents demonstrated the ability to make novel, meaningful discoveries in two exploratory tasks without oracle papers, with findings that matched concurrent research published after the models' knowledge cutoffs.
This research provides empirical evidence that environment design—specifically fostering decentralized, sustained, and reflective exploration—can mitigate the 'streetlight effect' in AI agents. While a gap remains between AI and human research capabilities, the results suggest that move away from centralized, metric-driven control towards more autonomous, self-regulating systems is a promising path for AI-driven scientific discovery.
Recent AI systems have made rapid progress in scientific discovery when given well-defined metrics, but whether they can autonomously undertake open-ended scientific discovery remains unclear. We investigate AI's ability to tackle open-ended tasks in Station, an open-world environment in which multiple agents simulate a scientific ecosystem. To tackle challenges specific to open-ended tasks, we propose augmenting Station with two mechanisms: a Supervisor mechanism and periodic Meta Reflection, which encourage persistent exploration even when intermediate metrics are lacking. We construct open-ended tasks from three recent oral papers presented at ICLR. We give agents the main research question studied in each paper while withholding the paper's results and disabling web access. We then measure how many of the original findings-partitioned into individual criteria-agents rediscover. We find that Station rediscovers 62.7% of the criteria on average, compared with 15.4% for Codex Multiagent-v2 and 14.4-20.6% for AI Scientist-v2. Ablation and behavioral analyses indicate that adding the two mechanisms together improves research coverage and continuity. We further evaluate Station on two open-ended tasks without oracle papers and find that some of the discoveries made by the agents closely match discoveries reported by researchers after the knowledge cutoff date. Together, these results indicate that a suitable environment can enable agents to autonomously make meaningful progress in open-ended scientific discovery.
Sam: They add a Supervisor that enforces commitment to each agent’s own proposal. It cannot choose research directions or run experiments. Its narrow job is to stop premature pivots, except when continuing is genuinely impossible.
Alex: That’s persistence without a central research director. But staying with a bad question longer could make the judgment problem worse.
Sam: That’s where Meta Reflection comes in: periodic reassessment of research direction and value. Agents adopt the perspective of an external human expert and critique their work. These reflection responses use the same language model, regardless of which model normally powers the agent.
Alex: What does that reassessment actually change? A concrete example would help distinguish it from simply asking the agent to think harder.
Sam: One agent studied how a puzzle-solving model updated information about static walls and corridors. Reflection prompted it to recognize the weak connection to planning. It then applied the same framework to movable boxes, asking how their changing positions affected the model’s plans.
Alex: It kept the analytical tool but changed what the tool was explaining. That sounds closer to connecting work with the main question than starting over.
Sam: That’s how the authors interpret it. The environment also preserves experiments, discussions, and agent-written papers for others to use. At the end, a separate consolidation agent gathers the findings and evidence into a research report.
Alex: Now the hard evaluation question: if scientific value has no clear score, how do they measure whether Station succeeds?
Sam: They use existing research as a reference. Each “oracle paper” is a paper whose findings are withheld from the agents. Agents receive its main question and the necessary experimental setup, with internet access disabled and no human intervention during the run.
Alex: That makes the task open-ended while the work happens, but assessable afterward. What research questions did they choose?
Sam: There are three rediscovery tasks, each repeated across independent runs. One asks whether reinforcement-learning agents—agents trained through rewards—develop internal plans. Another asks whether language-model outputs have a compact underlying structure. The last examines how recurrent neural networks, which carry information across steps, represent time and forget.
Alex: And recovering a finding means more than writing something that resembles the original paper’s conclusion?
Sam: It must be substantively similar and supported by experimental evidence. The authors divided the original findings into criteria before running Station. Automated evaluations were checked against a blinded expert assessment, with high agreement, although that check covered sampled reports from the planning task.
Alex: What’s the headline comparison, and were the competing systems given comparable opportunities to experiment?
Sam: Station recovered about 63 percent of the criteria on average. The competing systems recovered roughly 15 to 21 percent. They received the same questions and setups, and their cumulative experiment time was matched to meet or exceed Station’s task-specific budget.
Alex: That’s a large gap, but it doesn’t tell us where the reasoning gets difficult. Which findings were still out of reach?
Sam: In planning, Station was strongest at detecting internal representations of future actions or states. It was weaker at manipulating those representations and explaining planning mechanisms. It missed the oracle paper’s finding that planning extends both forward from boxes and backward from targets.
Alex: So detecting information inside a model was easier than explaining how the model uses it. Did that pattern recur?
Sam: In the language-output task, Station found compact structure and relationships that transferred across continuations. It recovered much less about using those relationships for prediction or generation. In the memory task, agents often optimized recall performance instead of explaining the mechanisms of representation and forgetting.
Alex: That sounds like the streetlight effect surviving the intervention. Do they show that commitment and reflection nevertheless improve performance?
Sam: Removing both mechanisms reduced planning-task recovery from about 76 percent to about 58 percent. The largest decline concerned causal interventions, which required building on earlier findings. Agents also spent less time on each research direction, supporting the authors’ interpretation of reduced persistence.
Alex: But removing both together doesn’t reveal which one mattered most. And the bigger comparison changes models and workflows as well as coordination.
Sam: The authors acknowledge both issues. They evaluate Station as a complete system, not as proof that decentralization alone explains the gains. Another limitation is possible training exposure to some oracle findings. Withholding papers and requiring experiments helps, but cannot rule that exposure out.
Alex: Rediscovery is useful, but new science is the larger claim. What evidence goes beyond those reference papers?
Sam: Exploratory tasks produced findings matching work released after the models’ knowledge cutoffs. One studied behavioral traits transferring through unrelated training data. Agents found that restricting LoRA—fine-tuning through low-rank weight updates—to early layers could restore failed transfer. The authors validated that finding across other traits.
Alex: Were those discoveries universal rules, or did follow-up testing narrow their scope?
Sam: The scope matters. A separate finding about adapter rank depended on training conditions in follow-up work. In vision-language models, which process images and text, reducing late-layer outputs improved visual-hallucination accuracy across tested models, but the benefit varied. Neither result establishes a universal recipe.
Alex: And research judgment remains the bottleneck. An author of the planning reference paper rated three-quarters of the non-overlapping agent proposals poorly. Who should read the full document, given both the gains and those gaps?
Sam: Builders and evaluators of autonomous research systems should start with Methods and Results on Rediscovery Tasks. Look closely at the combined ablation, then read Limitations before attributing the advantage to any single design choice. The released dialogues offer a way to inspect how findings developed.
Alex: For researchers who aren’t building these systems, what’s the one distinction to carry away?
Sam: In these tasks, organizing exploration helped agents produce supported findings. It did not give them reliable scientific taste. The line to remember is: finding an answer and choosing a question worth answering are different research skills.
Alex: Keep that distinction in mind when you read the next claim of autonomous discovery.