PRM-as-a-Judge Team
5 min
Evaluating embodied AI models has traditionally relied on binary task success rates or manual rule-based scores. However, these coarse metrics reduce complex robotic trajectories to simple yes-or-no outcomes, ignoring fine-grained execution quality. For instance, failed rollouts can vary dramatically—some fail early, while others complete nearly all subtasks before failing. Similarly, successful rollouts may be smooth or riddled with hesitant, inefficient corrections.
To address this limitation, the authors present PRM-as-a-Judge 1.5, an open-source assessment toolkit that upgrades progress-curve evaluation into a comprehensive, doctor-style diagnostic framework. The system translates raw robot rollout videos into dense progress curves using Process Reward Models (PRMs), computes multi-dimensional metrics, and generates detailed model assessment reports.
PRM-as-a-Judge 1.5 builds upon the Outcome-Process-Diagnosis (OPD) metric framework, incorporating three new conditioned metrics to capture nuanced execution behaviors:
To provide deeper insights into model capabilities, version 1.5 introduces three specialized metrics: Failure Near-Success (FNS) to measure how close failed rollouts came to completion, Drawdown Recovery Ratio (DRR) to quantify how well models recover from setbacks, and Success Quality Score (SQS) to evaluate the stability and smoothness of successful executions.
Evaluating mainstream Vision-Language-Action (VLA) models and World Action Models (WAMs) on the RoboDojo benchmark yields several surprising insights that contradict traditional leaderboard rankings. First, VLAs consistently outperform WAMs across nearly all metrics, holding a dominant representation in top-tier performance tiers. Second, model size does not guarantee superior performance; several smaller models outperform significantly larger counterparts, demonstrating a lack of positive correlation between parameter count and embodied manipulation capability. Finally, fine-grained process metrics reveal that models frequently trade off stability, recovery, and efficiency in ways that binary success rates completely obscure.
Fine-grained robotic evaluation matters for understanding embodied models, going beyond binary success rates and rule-based process scores. We present PRM-as-a-Judge 1.5, a toolkit for robot process assessment that turns rollout videos into dense progress curves and derives multiple fine metrics. PRM-as-a-Judge 1.5 introduces three metrics, building on version 1.0, that characterize failure-side progress, post-drawdown recovery, and success-side execution quality, helping users understand embodied model capability. Based on the rollout videos from benchmarks, we perform a comprehensive assessment of the embodied models, providing some fine-grained metric results and key findings. We further introduce RoboPulse++ to evaluate the reliability of process reward models (PRM), providing evaluators with a more accurate testing platform. Moreover, we release a user-friendly assessment suite, including the benchmark, metric implementation, and visualization tools, to support reproducible manipulation process evaluation. We call on the community to rethink how robots are evaluated and establish transparent, procedural, and reproducible assessment as a foundation for the next generation of embodied intelligence.
Sam: So a robot that looks good on a simple pass-fail test might not look as good under closer scrutiny?
Alex: Precisely. A robot might have a decent success rate overall, but when you look at the process metrics, you find it's frequently stalling, rarely recovering from setbacks, and finishing tasks in a messy, unreliable way. Another robot with a slightly lower success rate might actually be executing tasks more smoothly and consistently. The headline number hides that distinction entirely.
Sam: What about the gap between simulation and the real world? Did robots that performed well in virtual tests hold up physically?
Alex: Not reliably. The correlation between simulated performance and real-world performance was, in the paper's own assessment, mid-to-low. And every model tested showed a drop when moving from simulation to physical deployment. The paper suggests this is because simulations can't fully capture the messiness of physical reality — the friction of a surface, the slight unpredictability of how objects respond to being touched. Those details matter enormously when a robot is actually handling something.
Sam: So even a well-trained robot is essentially encountering a slightly different world the moment it leaves the simulator.
Alex: That's the implication, yes. And this connects to another finding. The system found that detecting regression — moments when a robot actually undoes its own progress, like knocking something over that it had just carefully placed — is significantly harder to evaluate than detecting forward progress. The evaluator models struggled more with those backward steps.
Sam: Why would regression be harder to spot?
Alex: The paper points to where errors tend to cluster. Most mistakes happen at the moment of physical interaction — picking something up, placing it down, making contact with an object. Those are the moments where things go wrong. The robots generally understand the logic of the task. They know what order to do things in. What trips them up is the physical execution, the precise contact dynamics that are genuinely difficult to model or evaluate.
Sam: So the bottleneck isn't strategic understanding — it's physical competence.
Alex: That's the core finding. And it's a useful distinction, because it tells researchers exactly where to focus. Rather than redesigning how robots plan tasks, the evidence here points toward improving how they handle physical interaction — and how we measure that handling in the first place.
Sam: It does seem like a more useful tool than a simple pass-fail score.
Alex: Significantly more useful. The goal is to shift robot evaluation from a verdict — pass or fail — to something closer to a diagnostic report. One that tells you not just whether the robot succeeded, but how it moved through the task, where it struggled, and what kind of failure it had. That's the kind of information that can actually drive improvement.
Sam: A much clearer picture of what's working and what isn't.
Alex: Exactly. And that clarity is ultimately what makes evaluation worth doing in the first place. Thanks for listening to ResearchPod.