Wenbo Huang, Jianxin Bi, Shunyu Yang, Ruiyang Si, Rui Ni, Qiang Wang, Shulong Jiang, Yintianrun Zhang, Feixiang Ruan, Kewei Zhang, Ye Huang, Shukai Gong, Daquan Zhou
7 min
Abstract
Self-improving embodied agents are increasingly being developed and compared on robot simulation benchmarks. Several recent works expose the benchmark's success criteria to the learning agent, a practice we call Criteria Exposure (CE). These criteria, originally written to verify policies trained by imitation learning, often demand more than the corresponding language instructions, so a score from such a verifier may not faithfully reflect whether the instructed task was completed, especially for robot-control programs written by an agent. As embodied coding agents report ever higher benchmark scores, a natural question arises: when a coding agent is exposed to the benchmark's criteria, does it generate code that better performs the instructed task, or code that simply overfits to the verification criteria? We conduct controlled experiments with a self-improving agent with and without CE, and find that with CE the agent quickly produces code with a higher success rate under the given criteria, but this advantage disappears when the criteria are changed slightly without departing from the original instructions. More importantly, the skills the agent develops under CE transfer poorly across tasks. We further propose instruction-faithful criteria and show that they better reflect the performance of agent-generated programs.
Alex: What is actually learning here? Are they updating the model controlling the robot?
Sam: No. The actor, which writes robot-control code, stays frozen. A separate learner reviews its attempts and revises a library: reusable control functions plus written advice. That library is what improves across rounds.
Alex: And the experimental switch is what the learner gets to see, rather than what the robot itself can sense?
Sam: Yes. Both settings receive trajectories and whether each attempt succeeded. The exposed learner also receives criteria code, scores, and completed stages. The actor never reads those directly; exposure reaches its behavior through the revised library.
Alex: That matters for interpretation. This tests a bundle of grading information, not source-code access in isolation.
Sam: Correct. The actor, robot tools, development layouts, and learning budget otherwise stay the same. After learning, the library becomes read-only, and the actor is tested on unseen layouts.
Alex: Before changing the grader, does ordinary success feedback produce useful learning?
Sam: It does. Across the learned tasks, success-only feedback substantially improves performance over an empty library. Exposure adds a further gain overall, but that gain comes from tasks where grading and instruction disagree. Where they agree, the settings perform similarly.
Alex: So the decisive test concerns the mismatched tasks. How did the authors change the rules without simply making success easier?
Sam: They rewrote criteria to track the instruction more closely. They removed hidden requirements, relaxed some thresholds, fixed defects, and tightened checks that overlooked instructed actions. Crucially, they judged the same recorded behavior under both rule sets.
Alex: That holds behavior constant. If the verdict changes, it’s the measurement changing, not a robot having a better day.
Sam: On the sixteen mismatched tasks, each library produced three hundred and twenty test episodes. Under the original rules, exposure achieved about fifty-two percent success. Success-only learning achieved about twenty-eight percent.
Alex: That’s a large apparent advantage. What remained when the same attempts were judged against the rewritten rules?
Sam: The exposed library scored forty-five percent. The success-only library scored about forty-seven percent. The authors describe that difference as within noise: the exposure advantage had disappeared.
Alex: The button example helps explain why. A library can learn the hidden confirmation sequence so faithfully that it violates the instruction’s sequence.
Sam: And that happens in the study. The exposed button library passes the original verifier but fails the instruction-faithful version. The success-only library shows almost the reverse pattern. Across tasks, though, not every rewrite changes scores; defect fixes and relaxed thresholds often leave them similar.
Alex: Could the replacement grader just favor whatever the success-only learner happens to do?
Sam: That’s the central caution. The replacement criteria are the authors’ own proxy for intended success. They manually checked episodes whose verdict changed and found no misjudgments under their new rules, but that isn’t independent validation.
Alex: Are there choices in those replacements that readers should inspect themselves?
Sam: Yes. The authors decide what counts as a tower and how tightly a nut must be seated. They also retain returning both arms home, even though many instructions omit it. So “instruction-faithful” describes their intended repair, not an unquestionable standard.
Alex: Did they also reverse the experiment—let the learner read the replacement rules and test it against the originals?
Sam: They did, on a subset of tasks. Each exposed library performed best under the criteria it had read. That supports the claim that scores follow the verifier, although the reverse test alone cannot establish which verifier is right.
Alex: What about useful skills underneath those scores? If exposure teaches better manipulation, some benefit ought to travel to new tasks.
Sam: They distilled source-task experience into shared libraries and tested unfamiliar RoboDojo tasks. Both libraries helped relative to no library, but the exposed library’s earlier lead did not return. The success-only library did better in that comparison.
Alex: And moving beyond RoboDojo would test whether that experience is more generally useful.
Sam: On LIBERO-PRO and RoboCasa365, neither transferred library gave a clear gain. Experience collected within RoboCasa365 helped more. The authors link this to different bottlenecks: tabletop experience lacked guidance for moving the robot’s base and operating kitchen fixtures.
Alex: That doesn’t show that exposure caused the transfer failure, then. Both kinds of imported experience struggled.
Sam: Nor are the proposed mechanisms settled. The authors infer them from written entries, skill calls, and actions, rather than experiments removing individual entries. Samples are small, distillation variability wasn’t estimated, and only RoboDojo received rewritten criteria.
Alex: Given those limits, who should read the full document, and where should they start?
Sam: Researchers building self-improving robot agents, and anyone designing their benchmarks. Start with “Changing the Criteria: Task or Verifier?” Focus on the same-episode comparison, then inspect the appendix’s criteria changes and the limitations. The evidence is strongest for a measurement problem in this particular loop, not a universal rejection of detailed feedback.
Alex: And for someone outside robot learning, what is the line to carry into their own evaluations?
Sam: An advantage from reading the grader is not necessarily an advantage at doing the task.
Alex: Thanks for listening.