Self-improving embodied agents are increasingly being developed and compared on robot simulation benchmarks. Several recent works expose the benchmark's success criteria to the learning agent, a practice we call Criteria Exposure (CE). These criteria, originally written to verify policies trained by imitation learning, often demand more than the corresponding language instructions, so a score from such a verifier may not faithfully reflect whether the instructed task was completed, especially for robot-control programs written by an agent. As embodied coding agents report ever higher benchmark scores, a natural question arises: when a coding agent is exposed to the benchmark's criteria, does it generate code that better performs the instructed task, or code that simply overfits to the verification criteria? We conduct controlled experiments with a self-improving agent with and without CE, and find that with CE the agent quickly produces code with a higher success rate under the given criteria, but this advantage disappears when the criteria are changed slightly without departing from the original instructions. More importantly, the skills the agent develops under CE transfer poorly across tasks. We further propose instruction-faithful criteria and show that they better reflect the performance of agent-generated programs.
Sam: A robot can get better at passing a test without getting better at following instructions. This study found that letting its learning system read the grading rules raised scores, but the advantage vanished under rewritten rules.
Alex: That sounds like a warning for anyone claiming their agent improves with experience. Whose work is this, and what kind of evidence are we discussing?
Sam: It’s a controlled simulation study, released as a preprint, by Wenbo Huang and colleagues at Peking University and the National University of Singapore. The title is “Learning the Task or Learning the Verifier? Rethinking Criteria Exposure in Embodied Self-Improvement.”
Alex: So the stakes are about what a higher score actually measures, not just whether a robot can complete a benchmark.
Sam: For researchers building robot agents or evaluating them, that distinction changes what counts as progress. We’ll unpack the comparison, the evidence that the score follows the grading rules, and where that conclusion needs caution.
Alex: Start with the mismatch. Why would the grading rules ask for something different from the instruction?
Sam: These benchmarks were originally built for robot policies trained by copying demonstrations. The demonstrations already contain details such as movement order and final positions. A code-writing agent, receiving only a language instruction, may never be told those details.
Alex: Can you make that concrete? A hidden requirement might be sensible, even if someone forgot to write it down.
Sam: In RoboDojo’s button task, the instruction says to press red buttons according to number cards, then press blue to confirm. But the original verifier—the code deciding success—requires a different sequence. It wants blue pressed between the red-button groups, and again afterward.
Alex: Following the instruction could therefore fail the benchmark. Reading the grading code would reveal the missing protocol, but that wouldn’t establish better instruction-following.
Sam: That’s the distinction the authors test. They call access to those rules “Criteria Exposure.” Existing systems provide anything from rule descriptions to the verifier’s source code, but usually treat that choice as part of the system design.
Alex: Researchers already know agents can exploit evaluators. What does this study add beyond another example of that problem?
Sam: It makes exposure the experimental variable within a fixed learning loop. Then it asks whether the extra score survives a different judgment of the same behavior. The authors also test whether the resulting experience helps on unfamiliar tasks.
Alex: What is actually learning here? Are they updating the model controlling the robot?
Sam: No. The actor, which writes robot-control code, stays frozen. A separate learner reviews its attempts and revises a library: reusable control functions plus written advice. That library is what improves across rounds.
Alex: And the experimental switch is what the learner gets to see, rather than what the robot itself can sense?
Sam: Yes. Both settings receive trajectories and whether each attempt succeeded. The exposed learner also receives criteria code, scores, and completed stages. The actor never reads those directly; exposure reaches its behavior through the revised library.
Alex: That matters for interpretation. This tests a bundle of grading information, not source-code access in isolation.
Sam: Correct. The actor, robot tools, development layouts, and learning budget otherwise stay the same. After learning, the library becomes read-only, and the actor is tested on unseen layouts.
Alex: Before changing the grader, does ordinary success feedback produce useful learning?
Sam: It does. Across the learned tasks, success-only feedback substantially improves performance over an empty library. Exposure adds a further gain overall, but that gain comes from tasks where grading and instruction disagree. Where they agree, the settings perform similarly.
Alex: So the decisive test concerns the mismatched tasks. How did the authors change the rules without simply making success easier?
Sam: They rewrote criteria to track the instruction more closely. They removed hidden requirements, relaxed some thresholds, fixed defects, and tightened checks that overlooked instructed actions. Crucially, they judged the same recorded behavior under both rule sets.
Alex: That holds behavior constant. If the verdict changes, it’s the measurement changing, not a robot having a better day.
Sam: On the sixteen mismatched tasks, each library produced three hundred and twenty test episodes. Under the original rules, exposure achieved about fifty-two percent success. Success-only learning achieved about twenty-eight percent.
Alex: That’s a large apparent advantage. What remained when the same attempts were judged against the rewritten rules?
Sam: The exposed library scored forty-five percent. The success-only library scored about forty-seven percent. The authors describe that difference as within noise: the exposure advantage had disappeared.
Alex: The button example helps explain why. A library can learn the hidden confirmation sequence so faithfully that it violates the instruction’s sequence.
Sam: And that happens in the study. The exposed button library passes the original verifier but fails the instruction-faithful version. The success-only library shows almost the reverse pattern. Across tasks, though, not every rewrite changes scores; defect fixes and relaxed thresholds often leave them similar.
Alex: Could the replacement grader just favor whatever the success-only learner happens to do?
Sam: That’s the central caution. The replacement criteria are the authors’ own proxy for intended success. They manually checked episodes whose verdict changed and found no misjudgments under their new rules, but that isn’t independent validation.
Alex: Are there choices in those replacements that readers should inspect themselves?
Sam: Yes. The authors decide what counts as a tower and how tightly a nut must be seated. They also retain returning both arms home, even though many instructions omit it. So “instruction-faithful” describes their intended repair, not an unquestionable standard.
Alex: Did they also reverse the experiment—let the learner read the replacement rules and test it against the originals?
Sam: They did, on a subset of tasks. Each exposed library performed best under the criteria it had read. That supports the claim that scores follow the verifier, although the reverse test alone cannot establish which verifier is right.
Alex: What about useful skills underneath those scores? If exposure teaches better manipulation, some benefit ought to travel to new tasks.
Sam: They distilled source-task experience into shared libraries and tested unfamiliar RoboDojo tasks. Both libraries helped relative to no library, but the exposed library’s earlier lead did not return. The success-only library did better in that comparison.
Alex: And moving beyond RoboDojo would test whether that experience is more generally useful.
Sam: On LIBERO-PRO and RoboCasa365, neither transferred library gave a clear gain. Experience collected within RoboCasa365 helped more. The authors link this to different bottlenecks: tabletop experience lacked guidance for moving the robot’s base and operating kitchen fixtures.
Alex: That doesn’t show that exposure caused the transfer failure, then. Both kinds of imported experience struggled.
Sam: Nor are the proposed mechanisms settled. The authors infer them from written entries, skill calls, and actions, rather than experiments removing individual entries. Samples are small, distillation variability wasn’t estimated, and only RoboDojo received rewritten criteria.
Alex: Given those limits, who should read the full document, and where should they start?
Sam: Researchers building self-improving robot agents, and anyone designing their benchmarks. Start with “Changing the Criteria: Task or Verifier?” Focus on the same-episode comparison, then inspect the appendix’s criteria changes and the limitations. The evidence is strongest for a measurement problem in this particular loop, not a universal rejection of detailed feedback.
Alex: And for someone outside robot learning, what is the line to carry into their own evaluations?
Sam: An advantage from reading the grader is not necessarily an advantage at doing the task.
Alex: Thanks for listening.