scBench-Long: Verifiable Benchmarking of Long-Horizon Single-Cell Biology

Single-cell biology requires researchers to integrate raw measurements with metadata, assay context, and external knowledge to form specific biological claims. While AI agents have shown promise in local analysis tasks, it remains unclear whether they can perform the long-horizon reasoning necessary to derive scientific conclusions from raw data. This paper introduces scBench-Long, a benchmark designed to evaluate whether AI agents can successfully navigate these complex, multi-step scientific workflows.

The authors constructed 21 distinct evaluations spanning five major study systems, including melanoma CD8 T-cell reactivity, CD8 RNA+ATAC regulatory inference, human-monkey chimera development, KRAS-driven lung tumor aging, and lethal COVID-19 lung pathology. Unlike existing benchmarks that focus on isolated tool execution, scBench-Long requires agents to process raw or near-raw data, integrate multiple modalities (e.g., scRNA-seq, scATAC-seq, TCR repertoires), and arrive at a verifiable scientific conclusion. The benchmark uses deterministic grading against controlled answer vocabularies, supplemented by rubric-based trajectory diagnostics to assess partial progress.

Across 1,068 completed trajectories, performance was generally low. The strongest model-harness pair, Claude Opus 4.8 with Claude Code, passed 16 out of 63 runs (25.4%). The authors observed that agents often successfully performed individual analysis steps—such as identifying cell populations or reconstructing repertoires—but failed to synthesize these into the correct final scientific claim. A recurring failure mode was the tendency of models to rely on familiar biological priors (e.g., canonical markers) rather than the specific evidence provided in the dataset, often ignoring task-specific data in favor of generalized knowledge.