Closed-loop Auto Research for Molecular Property Prediction: Discovering and Certifying Generalizable Improvements

Research Approach

This paper investigates whether autonomous agents—using language models to edit machine learning pipelines—can improve molecular property prediction models in a way that generalizes to unseen data. The authors implement a 'closed-loop' system where agents propose interventions across three specific axes: molecular features, model architecture/code, and the acquisition of external experimental data. To ensure scientific rigor, they employ a 'file-level ablation lock,' which restricts each experimental trial to a single intervention axis. This allows the researchers to attribute performance gains directly to a specific type of research action.

Held-out Certification

A critical component of the study is the distinction between validation performance and held-out certification. Because agents can overfit to the validation signal—either through selection variance (noise) or distribution shift (when external data is used)—the authors freeze the best-performing configurations found during the search and evaluate them exactly once on a held-out test set that the agent never accessed. This 'certify-after-search' approach reveals that while agents consistently improve validation scores, these gains do not always translate to the test set, effectively identifying non-transferable signatures.

Findings and Impact

Across 36 endpoints in three benchmark suites (TDC, MoleculeNet, and Polaris), the agentic pipeline achieved positive held-out gains. Interestingly, the most effective research axis was regime-dependent: external data acquisition proved most valuable for TDC, model-code changes for Polaris, and a combination of feature and model interventions for MoleculeNet. The study demonstrates that these agentic interventions outperform standard automated machine learning (AutoML) controls, which failed to replicate the agent's code-level improvements. By separating the discovery phase from a strict held-out certification, the authors provide a domain-agnostic framework for evaluating any autonomous system that optimizes a proxy for a held-out quantity.