Fine-grained robotic evaluation matters for understanding embodied models, going beyond binary success rates and rule-based process scores. We present PRM-as-a-Judge 1.5, a toolkit for robot process assessment that turns rollout videos into dense progress curves and derives multiple fine metrics. PRM-as-a-Judge 1.5 introduces three metrics, building on version 1.0, that characterize failure-side progress, post-drawdown recovery, and success-side execution quality, helping users understand embodied model capability. Based on the rollout videos from benchmarks, we perform a comprehensive assessment of the embodied models, providing some fine-grained metric results and key findings. We further introduce RoboPulse++ to evaluate the reliability of process reward models (PRM), providing evaluators with a more accurate testing platform. Moreover, we release a user-friendly assessment suite, including the benchmark, metric implementation, and visualization tools, to support reproducible manipulation process evaluation. We call on the community to rethink how robots are evaluated and establish transparent, procedural, and reproducible assessment as a foundation for the next generation of embodied intelligence.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a new framework for evaluating robots — one that goes well beyond simply asking whether a robot succeeded or failed.
Sam: So the core problem is that standard robot tests just give you a thumbs up or thumbs down?
Alex: Exactly. And that binary view leaves researchers almost completely in the dark. Did the robot fail immediately, or did it make it ninety percent of the way through before dropping the object? A simple pass-fail score treats both situations identically.
Sam: Which seems like a real problem if you're trying to figure out what actually went wrong.
Alex: It is. Think of it like grading a gymnastics routine by only noting whether the gymnast landed the final jump. You'd miss everything — the wobbles, the mid-air corrections, the near-falls. The researchers behind this paper argue that robot evaluation needs the equivalent of a full judging panel, not just a finish-line camera.
Sam: So what's their solution?
Alex: They built a system called PRM-as-a-Judge 1.5. The name is a bit technical, but the idea is straightforward. Instead of watching a robot's performance and asking "did it finish?", the system watches the entire video of the robot working and estimates, at every moment, how far along the task the robot actually is. You end up with a curve — a kind of progress graph — rather than a single yes or no.
Sam: Like a live progress bar tracking the robot throughout the task.
Alex: That's a good way to put it. And because you have that full curve, you can ask much more precise questions. Did the robot stall out halfway through? Did it recover after a setback? Did it finish cleanly, or did it barely scrape across the line?
Sam: How do they actually measure those things?
Alex: They developed a set of specific metrics — each one designed to capture a different dimension of performance. One metric, for instance, looks at what they call near-success failures: cases where the robot made it almost to the end before something went wrong. Another tracks recovery — specifically, when a robot loses progress, how much of that lost ground does it manage to win back? These aren't just interesting observations. They're quantifiable scores that can be compared across different robots and different tasks.
Sam: Did they actually test this on real robots?
Alex: They did, across a benchmark called RoboDojo, which includes both computer simulations and physical real-world tasks. And the results were instructive. When they compared the standard success-rate rankings to the new process-level rankings, they didn't always match up.
Sam: So a robot that looks good on a simple pass-fail test might not look as good under closer scrutiny?
Alex: Precisely. A robot might have a decent success rate overall, but when you look at the process metrics, you find it's frequently stalling, rarely recovering from setbacks, and finishing tasks in a messy, unreliable way. Another robot with a slightly lower success rate might actually be executing tasks more smoothly and consistently. The headline number hides that distinction entirely.
Sam: What about the gap between simulation and the real world? Did robots that performed well in virtual tests hold up physically?
Alex: Not reliably. The correlation between simulated performance and real-world performance was, in the paper's own assessment, mid-to-low. And every model tested showed a drop when moving from simulation to physical deployment. The paper suggests this is because simulations can't fully capture the messiness of physical reality — the friction of a surface, the slight unpredictability of how objects respond to being touched. Those details matter enormously when a robot is actually handling something.
Sam: So even a well-trained robot is essentially encountering a slightly different world the moment it leaves the simulator.
Alex: That's the implication, yes. And this connects to another finding. The system found that detecting regression — moments when a robot actually undoes its own progress, like knocking something over that it had just carefully placed — is significantly harder to evaluate than detecting forward progress. The evaluator models struggled more with those backward steps.
Sam: Why would regression be harder to spot?
Alex: The paper points to where errors tend to cluster. Most mistakes happen at the moment of physical interaction — picking something up, placing it down, making contact with an object. Those are the moments where things go wrong. The robots generally understand the logic of the task. They know what order to do things in. What trips them up is the physical execution, the precise contact dynamics that are genuinely difficult to model or evaluate.
Sam: So the bottleneck isn't strategic understanding — it's physical competence.
Alex: That's the core finding. And it's a useful distinction, because it tells researchers exactly where to focus. Rather than redesigning how robots plan tasks, the evidence here points toward improving how they handle physical interaction — and how we measure that handling in the first place.
Sam: It does seem like a more useful tool than a simple pass-fail score.
Alex: Significantly more useful. The goal is to shift robot evaluation from a verdict — pass or fail — to something closer to a diagnostic report. One that tells you not just whether the robot succeeded, but how it moved through the task, where it struggled, and what kind of failure it had. That's the kind of information that can actually drive improvement.
Sam: A much clearer picture of what's working and what isn't.
Alex: Exactly. And that clarity is ultimately what makes evaluation worth doing in the first place. Thanks for listening to ResearchPod.