Measuring Biological Capabilities and Risks of AI Agents: Generating and Interpreting Evidence from Agentic Evaluations

As AI systems evolve from static models into agentic systems capable of autonomous, multi-step scientific research, they introduce significant new biosecurity challenges. These AI scientists can potentially accelerate the development of biological weapons by automating complex tasks across the design-build-test-learn cycle. While traditional benchmarks measure a model's ability to answer isolated questions, they fail to capture how an agent might synthesize information, plan experiments, or interact with laboratory tools to cause harm.

Agentic evaluations are designed to fill this gap by testing how AI systems navigate complex, interdependent workflows. Unlike static tests, these evaluations measure an agent's ability to plan, adapt, and use tools in environments that mirror real-world scientific research. However, the authors emphasize that these evaluations are not 'plug-and-play' metrics. Because they require complex setup, the results are deeply influenced by the specific assumptions made by the researchers who design them.

To ensure that evaluation results are meaningful, the authors outline several key areas where design choices significantly impact outcomes:

• Task Definition: Evaluations must be grounded in cross-disciplinary expertise, combining biology, machine learning, and biosecurity to ensure that the tasks are technically plausible and relevant to real-world threats.
• Operational Context: The environment in which an agent operates—including the tools provided, the level of human oversight, and the available resources—can drastically change the agent's performance.
• Scoring and Documentation: How success is defined and how the evaluation process is documented are often opaque. The authors argue that without transparent reporting on these choices, it is impossible for policymakers to accurately assess the true risk posed by an AI system.