PRM-as-a-Judge Team
5 min
Abstract
Fine-grained robotic evaluation matters for understanding embodied models, going beyond binary success rates and rule-based process scores. We present PRM-as-a-Judge 1.5, a toolkit for robot process assessment that turns rollout videos into dense progress curves and derives multiple fine metrics. PRM-as-a-Judge 1.5 introduces three metrics, building on version 1.0, that characterize failure-side progress, post-drawdown recovery, and success-side execution quality, helping users understand embodied model capability. Based on the rollout videos from benchmarks, we perform a comprehensive assessment of the embodied models, providing some fine-grained metric results and key findings. We further introduce RoboPulse++ to evaluate the reliability of process reward models (PRM), providing evaluators with a more accurate testing platform. Moreover, we release a user-friendly assessment suite, including the benchmark, metric implementation, and visualization tools, to support reproducible manipulation process evaluation. We call on the community to rethink how robots are evaluated and establish transparent, procedural, and reproducible assessment as a foundation for the next generation of embodied intelligence.
Sam: So a robot that looks good on a simple pass-fail test might not look as good under closer scrutiny?
Alex: Precisely. A robot might have a decent success rate overall, but when you look at the process metrics, you find it's frequently stalling, rarely recovering from setbacks, and finishing tasks in a messy, unreliable way. Another robot with a slightly lower success rate might actually be executing tasks more smoothly and consistently. The headline number hides that distinction entirely.
Sam: What about the gap between simulation and the real world? Did robots that performed well in virtual tests hold up physically?
Alex: Not reliably. The correlation between simulated performance and real-world performance was, in the paper's own assessment, mid-to-low. And every model tested showed a drop when moving from simulation to physical deployment. The paper suggests this is because simulations can't fully capture the messiness of physical reality — the friction of a surface, the slight unpredictability of how objects respond to being touched. Those details matter enormously when a robot is actually handling something.
Sam: So even a well-trained robot is essentially encountering a slightly different world the moment it leaves the simulator.
Alex: That's the implication, yes. And this connects to another finding. The system found that detecting regression — moments when a robot actually undoes its own progress, like knocking something over that it had just carefully placed — is significantly harder to evaluate than detecting forward progress. The evaluator models struggled more with those backward steps.
Sam: Why would regression be harder to spot?
Alex: The paper points to where errors tend to cluster. Most mistakes happen at the moment of physical interaction — picking something up, placing it down, making contact with an object. Those are the moments where things go wrong. The robots generally understand the logic of the task. They know what order to do things in. What trips them up is the physical execution, the precise contact dynamics that are genuinely difficult to model or evaluate.
Sam: So the bottleneck isn't strategic understanding — it's physical competence.
Alex: That's the core finding. And it's a useful distinction, because it tells researchers exactly where to focus. Rather than redesigning how robots plan tasks, the evidence here points toward improving how they handle physical interaction — and how we measure that handling in the first place.
Sam: It does seem like a more useful tool than a simple pass-fail score.
Alex: Significantly more useful. The goal is to shift robot evaluation from a verdict — pass or fail — to something closer to a diagnostic report. One that tells you not just whether the robot succeeded, but how it moved through the task, where it struggled, and what kind of failure it had. That's the kind of information that can actually drive improvement.
Sam: A much clearer picture of what's working and what isn't.
Alex: Exactly. And that clarity is ultimately what makes evaluation worth doing in the first place. Thanks for listening to ResearchPod.