Imaging-101: Benchmarking LLM Coding Agents on Scientific Computational Imaging

Research Motivation

Computational imaging is essential for modern science, enabling the recovery of hidden signals from indirect, noisy measurements. However, developing these pipelines is notoriously difficult, requiring deep domain expertise in physics, mathematics, and numerical programming. While Large Language Models (LLMs) have shown promise in general coding tasks, their ability to handle the specialized, multi-stage requirements of scientific reconstruction—where a single error in physics modeling can invalidate the entire result—remains largely untested.

The Imaging-101 Benchmark

To address this, the authors introduce Imaging-101, a benchmark comprising 57 expert-verified computational imaging tasks across six scientific domains, including astronomy, medicine, and earth science. Each task is grounded in a peer-reviewed publication and canonicalized into a standardized four-stage pipeline: preprocessing, forward physics modeling, inverse solver, and visualization. This structure allows for consistent evaluation across three distinct tracks: planning (designing the approach), function-level unit tests (implementing specific modules), and end-to-end reconstruction (delivering a full, numerically verified pipeline).

Key Findings

Evaluating seven frontier LLMs, the study identifies systematic failure modes that generic coding benchmarks fail to capture. The models frequently struggle with scientific reasoning, such as omitting critical unit conversions, selecting inappropriate mathematical solvers for specific physical constraints, or failing to interpret numerical residuals as signs of incorrect conventions. These findings suggest that current LLMs lack the domain-specific physical intuition required for reliable scientific assistance, pointing toward the need for specialized, skill-augmented agents.

This work provides a rigorous framework for assessing AI in scientific discovery. By moving beyond simple code generation to focus on the numerical and physical correctness of scientific pipelines, Imaging-101 offers a clear path for developing research copilots that can reliably assist scientists in complex data reconstruction tasks.