Yuming Fu, Weijia Wu, Jing Chen, Jiahao Tang, Feifei Chen, Hongyu Zhu, Xin Jin, Alex Jinpeng Wang
9 min
Abstract
Self-driving laboratories (SDLs) aim to automate the full experimental loop, from scientific decision-making to physical execution. Ideally, AI-generated plans could be carried out directly by robotic systems, but reliable automation remains difficult in complex, open-world laboratory environments, where experiments often involve fine-grained manipulation, long-horizon procedures, and substantial variation across tasks and setups. Humans therefore remain an important execution interface between AI-generated plans and physical experiments, creating a need for clear and effective human-facing experimental guidance. Because laboratory procedures are inherently visual, spatial, and dynamic, video is particularly well suited to communicating apparatus configurations, manipulation actions, temporal dependencies, and state changes. Recent advances in video generation now make it possible to synthesize experimental demonstrations directly from an initial workspace image and a natural-language instruction. However, whether such models can reliably communicate real laboratory procedures has not been systematically studied. We introduce LabInstruct, a benchmark for situated instructional video generation in real laboratories. LabInstruct contains 204 tasks across 5 scientific disciplines, with real reference executions and structured annotations of objects, actions, contacts, and state transitions. Evaluating 8 frontier image-to-video models, we find that visually plausible generations frequently remain procedurally incorrect, revealing a substantial gap between visual realism and the reliability required for experimental instruction.
Sam: They explicitly try not to. The reference is one valid execution, not the only acceptable trajectory. Human reviewers remove incidental details, such as hand choice or pauses, unless those details matter for correctness.
Alex: Give me a task where that distinction becomes concrete. What must be preserved, and what can vary?
Sam: Take zeroing a digital scale with a weighing boat already on its pan. The demonstration must show the relevant button press and the display changing to zero. A different hand movement can be acceptable; pressing the button without showing the required display change is not enough.
Alex: So they separate the action from its consequence. A finger reaching the control doesn’t prove the measurement state changed.
Sam: They call that consequence a state transition: an observable change caused by an action. The annotations connect objects, contacts, ordered actions, and visible before-and-after states. Model-assisted drafts undergo human verification before the final instructions and evaluation checklists are frozen.
Alex: Let’s turn to scoring. A video might preserve every object beautifully while missing the essential operation. How do they stop appearance from dominating the result?
Sam: They assess object identity, action execution, resulting states, and physical plausibility separately, then combine them so weaknesses matter. A completion factor penalizes unmet critical requirements. Explicit workspace-continuity or visible-safety violations can cancel the overall score.
Alex: And what happens when the crucial moment is hidden behind the operator’s hand? For instruction, uncertainty itself seems like a problem, even if the action might have happened.
Sam: The checklist allows “yes,” “no,” or “unjudgeable.” In the core dimensions, insufficient visibility counts against the video. But uncertainty alone doesn’t trigger the scene or safety vetoes; those require an observed violation.
Alex: That’s a useful distinction: failure to demonstrate an action versus evidence of an unsafe action. Who made those judgments, and did people check the automated evaluator?
Sam: A video-understanding model judged sampled frames using the frozen checklists. Separately, human raters assessed every generated video, with disagreements resolved through expert adjudication. The human results broadly confirmed the main pattern: preserving appearance was easier than completing the requested procedure.
Alex: What is the headline result in numbers? And tell us what the number means, because a benchmark score isn’t automatically a success rate.
Sam: Under human evaluation, the leading model, MiniMax H3, scored about 44 out of a hundred overall. Seedance 2.0 followed at about 43. These are composite scores combining execution quality, completion, and validity—not percentages of procedures completed successfully.
Alex: Those leading scores are close. Was the automated judge equally strict, or could it make weak demonstrations look more usable than human reviewers thought?
Sam: It was generally more permissive, especially for weaker generators. For LTX, the automated overall score was about 22, compared with just three under human evaluation. Aggregate model comparisons agreed much better than judgments about individual tasks.
Alex: So automated evaluation can capture the broad performance structure without being interchangeable with expert review. What actually went wrong inside the generated videos?
Sam: The recurring failures were omitted actions, incorrect ordering, and manipulation of the wrong object. Other videos showed an action without its required consequence, such as a changed liquid level or instrument reading. Objects sometimes deformed, disappeared, or merged, although action errors were more common.
Alex: The benchmark also separates short steps from longer procedures. Does the drop on longer tasks show that duration itself causes the failures?
Sam: No. The levels contain independently curated task sets, not matched versions of identical tasks. The longer, dependent procedures generally expose greater completion problems, but the design doesn’t isolate length as the cause.
Alex: There’s another reason to be cautious about the leaderboard: video generation is variable. If a model gets another attempt, does the winner stay the same?
Sam: The main comparison uses a single generation per model and task. On a balanced subset of 40 tasks, the authors selected the best of several attempts. Seedance and Wan 3.0 gained more than MiniMax, changing the ordering within the leading group.
Alex: So the strongest group is more stable than the exact winner. How far beyond these selected laboratory clips should we carry the conclusion?
Sam: The dataset is deliberately clean, short, and visually verifiable, with uneven coverage and heavy reliance on one video platform. Possible training-data overlap cannot be ruled out. And the study evaluates videos, not whether people following them execute experiments correctly or safely.
Alex: With those boundaries in mind, who should read the full document, and where should they begin?
Sam: Researchers building video generators, scientific agents, or laboratory instruction tools should start with Evaluation Protocol, then Human Evaluation Details. Read Effect of Re-sampling before treating the leaderboard as definitive. The authors also stress that a visible-safety score cannot certify a procedure or replace expert supervision.
Alex: And for researchers who only need the central lesson?
Sam: A video can look like a demonstration without demonstrating the procedure.
Alex: Thanks for listening.