Andrew Perrault, Anmol Takiar, Dominik Winecki, Arnab Nandi
8 min
Abstract
Augmenting worker capabilities in the physical world through guidance has been a decades-long goal for AI agents. One key supporting capability is to understand what the worker is doing, seeing through the human's camera or smart glasses, and assist them in documentation. Despite advances in vision language models, their capabilities in these settings remain uncertain and are challenging to benchmark because most physical work data is proprietary, and involves too much variability to study in a principled manner. Unlike structured settings such as kitchens and warehouses that have predictable lighting, geometry, and objects, field-facing tasks such as rooftop solar installation and vehicle repair are unconstrained and high variance. Here, we model real-world physical work in Blender across eleven environments, incorporating variants in both image acceptability and documentation. The acceptability task asks whether a particular view would be suitable proof-of-work and documentation asks whether an agent can understand the task-relevant state of the environment from that view. Across 26 vision-language models, we find that top models largely solve documentation when the answer is text in frame, but are weak at reading measurements and at judging small parts and spatial relations, such as whether a tape measure leans. Acceptability is harder to pin down because human raters disagree with each other. Measured against a scorer that agrees with the raters' majority about as often as a single rater does, the best models cannot be distinguished from a single non-author rater, at a cost within the range of a human pass. In the three environments where we can trace acceptability errors, most wrong rejections come from misreading the scene rather than the requirement. On the three documentation questions for which we have real images, model rankings on rendered frames transfer to real images.
Sam: Give me an example where knowing the scene helps more than simply having a human caption.
Alex: Take a solar panel standing above a roof, with a tape measure beside it. The model must read where the panel frame crosses the tape, estimate the perpendicular gap, and say whether the tape leans. Those are distinct answers, and perspective can make the visible reading differ from the physical gap.
Sam: But the scene can tell you the geometry, not whether a human reviewer would accept the photograph. Isn’t that judgment still subjective?
Alex: It is. Their acceptability scorer checks conditions such as visibility, viewing angle, sharpness, and legibility. They adjust it using human ratings, then freeze it before evaluating agreement on held-out photographs.
Sam: How well does that reference match people? If reviewers disagree, the model’s score could partly reflect whose interpretation got written into code.
Alex: On held-out frames, the scorer agrees with the human majority about 82 percent of the time. That falls within the reported agreement range for individual raters against their colleagues. But the scorer is stricter than the majority, rejecting more photographs people would accept.
Sam: Do they force ambiguous requirements into a definite answer, or leave some cases unresolved?
Alex: They leave some unresolved. Borderline interpretations become gray zones and are excluded when they determine the verdict. They also drop license-plate acceptability because raters disagree about small but legible plates.
Sam: Now that we know what the reference means, how broad was the actual model test?
Alex: They evaluate 26 models. Each receives an image, a documentation requirement, and a request for structured answers. The final benchmark contains roughly fourteen hundred rendered frames, although acceptability and documentation use different subsets.
Sam: Synthetic photographs could be unusually clean. Do these include the capture problems somebody would encounter while working?
Alex: They vary phone-camera properties and introduce missed focus, hand shake, sensor noise, and compression. Documentation is mainly scored on photographs the reference considers acceptable. So even the clearer, adequately framed images can expose failures.
Sam: What separates the tasks models handle well from the ones they miss?
Alex: Most tested text-reading fields score highly across models. Long vehicle identification numbers and digital gauge displays remain exceptions. Small parts and relationships are harder: a bolt’s presence, a wire touching shingles, or a tape leaning.
Sam: Reading a tape sounds like reading digits. Does the benchmark actually show that it belongs in a different difficulty category?
Alex: Yes. Even the best models read the tape crossing within an eighth of an inch only 36 percent of the time. Unlike transcribing a label, this requires locating the crossing and interpreting the scale at that location.
Sam: And if a model rejects the photograph, how can they tell whether it misunderstood the instruction or simply saw the scene incorrectly?
Alex: They trace cases where the documentation answer overlaps the acceptance rule. On acceptable tape photographs, for example, the tape really is upright relative to the roof. Yet models call it tilted 75 percent of the time.
Sam: Could they be judging vertical against the picture’s edge instead of perpendicular against the roof?
Alex: That’s the authors’ interpretation. Tilt reports rise as the camera rolls, with close shots lacking roof context particularly difficult. Across the traced environments, models generally reject the defects they report; most wrong rejections accompany a scene misreading.
Sam: The prompt asks for the verdict first. Could a model reject the image, then invent a defect to justify its decision?
Alex: They test that by moving the verdict last and removing it altogether. For most models re-tested, defect reports stay within run-to-run variation. Some cheaper models report fewer defects without the verdict, so answer order does matter for those.
Sam: There’s also a human comparison. Should we hear that as models being ready to replace reviewers?
Alex: No. The strongest models cannot be distinguished from a single non-author rater against this scorer. But those raters are non-experts, and matching one noisy reviewer does not establish reliable field inspection.
Sam: Then the real-photograph check seems crucial. Does it validate the review task, or only the ability to extract information?
Alex: Only information extraction, on three matched questions: tape readings, the visible face of a car, and damage severity. Models ranking well on rendered images also tend to rank well on real photographs. That supports the benchmark’s diagnostic value, not real-world acceptability performance.
Sam: So who should read the full document, and where should they start?
Alex: Researchers building physical-work assistants or visual evaluation benchmarks should read it. Start with “Building the Benchmark Set and Acceptability Scorer” to understand the reference judgments. Then read “Analysis” for the scene-misreading evidence and the verdict-order test.
Sam: And for researchers outside that area, what’s the distinction worth carrying into their own evaluations?
Alex: Reading the words in a photograph is not the same as understanding the physical evidence. This benchmark makes that gap testable, while leaving real-world proof-of-work review an open question.
Sam: Keep that distinction in mind when a model says the photo isn’t good enough.