Zefan Wang, Yufan Dang, Xuanwei Shi, Mingqi Gao, Cheng Yin, SikYuen Tam, Hanyu Liu, Yasheng Wang, Yaxi Lu, Songyang Zhang, Xin Cong, Yuan Yao
7 min
Abstract
GPT-6 Astra can drive a robot arm directly through a fixed harness, and scores of such robot-use agents on RoboDojo have risen quickly since. Reported systems differ in harness and test subset and often change several designs at once, so it is hard to tell which choices produce the gains. We take these designs apart and test them in one framework, along the harness around the model, the action interface, the reuse of experience and the scene geometry given to the model. A lean harness that keeps the full episode history and stops each motion early goes a long way. Reusing experience changes how the model acts but is not a free lunch, and scene geometry helps most, even when depth is predicted from RGB alone. JustRUA starts from the official L3 harness and keeps only what matters most. Even with no learned policy, self-improvement loop or task-specific tuning, this simple harness raises the score on the RoboDojo simulation benchmark substantially, from 29.7 to 50.3.
Sam: Let's take the simplest changes first. What was wrong with the original action loop?
Alex: It retained only the most recent images, although earlier text stayed available. The revised loop kept the whole episode history, including images. That lets the model refer back to an object now hidden behind an arm, or the scene before contact.
Sam: Remembering helps interpretation. What stopped a bad movement once the robot had started it?
Alex: The authors capped each planned movement at twenty control steps, then returned fresh images and measured state. Any unfinished motion was discarded. The model could continue toward the target or change its approach.
Sam: So this didn't improve the planner itself. It shortened how long the robot could move before the model got another look.
Alex: That's the distinction. They had observed occasional unstable plans causing large, unreasonable movements. Earlier feedback limited how far those motions could go, and full history plus shorter motions improved the score substantially.
Sam: Did giving the model more natural commands help? Asking it to pick something up sounds easier than specifying coordinates.
Alex: They tested named skills, such as approach, transport, and handover. Those changed behavior without improving the overall score. Relative movements also lost to plain targets expressed in the world's coordinate system.
Sam: That pushes against the idea that a more elaborate interface must be easier to use. What about teaching through examples instead?
Alex: Demonstrations and a self-written skill library changed how the model acted, but neither clearly beat the default. In one tile-flipping task, demonstrations supplied a missing handover between arms. Yet the robot still failed during the final push into a row.
Sam: So fixing the plan didn't guarantee finishing the task. There was another failure waiting later in the sequence.
Alex: And the examples increased episode costs. They also came from the tasks being scored, which makes those results hard to compare with an evaluation without that experience. The authors excluded all experience reuse from JustRUA.
Sam: Then where did the largest gains come from, if not smarter-looking commands or extra practice?
Alex: Scene geometry: information about where things are in space. Compact gripper readings helped the model distinguish a requested closure from an actual grasp. Depth images, which encode distance from the camera, helped much more.
Sam: Was that depth privileged information from the simulator? That could make the comparison less useful for an ordinary camera setup.
Alex: They tested simulator depth as a reference, but kept depth predicted from ordinary color images. After adding predicted depth, empty grasps fell from forty-one to sixteen in the reviewed episodes. Those counts cover the first layout of each task.
Sam: Predicted distance can be wrong. Did the model need an accurate reconstruction of the scene?
Alex: Apparently not in these runs. Predicted distances could be off by tens of centimetres, yet their relative ordering was accurate in the frames examined. The authors suggest that knowing what lies nearer or farther may explain much of the benefit.
Sam: If geometry helped, did explicit suggestions about where to grasp help further?
Alex: No tested version did. Those suggestions are affordances: information about possible actions on objects. The authors supplied candidate grasps and reachable surfaces, but these additions neither improved grasp reliability nor beat depth from the same source.
Sam: That information also arrives when the model isn't choosing a grasp. So the system keeps paying for something it needs only occasionally.
Alex: That's the authors' argument. The full history is resent at every decision, so extra observations keep adding cost. Their final configuration was more than twice as expensive per episode as the simpler default, even using their optimistic cached-input accounting.
Sam: What accounts for the final jump? Was that another kind of scene information?
Alex: The model gained permission to run code, mostly to calculate coordinates from image pixels and camera calibration. In a tube-insertion example, it located tubes and rack holes geometrically. It completed a layout that predicted depth alone had left unfinished, using fewer decisions.
Sam: Can we attribute the final improvement to those calculations alone?
Alex: No. At the same time, the authors removed a cap on model decisions, while retaining the benchmark's movement limit. They explicitly report that as a combined change. More broadly, this is a sequence of additions, not a test of every possible combination.
Sam: So how far should we trust the overall conclusion?
Alex: I find it useful evidence about this setup, not a general ranking of robot designs. Each configuration ran once on the selected evaluation set. Everything used one model and one simulation benchmark, and these scores are not official leaderboard scores.
Sam: And the remaining failures weren't all one missing feature that another simple change could fix?
Alex: Most were task-specific: plans that didn't suit the arm, accidental knocks, and slipping grasps. The authors argue that resolving them would often require trial and error on individual test tasks. That approaches the tuning they deliberately avoided.
Sam: Who should read the full document, and where should they start?
Alex: Robot-agent builders should start with Harness and Action, then Scene Geometry, especially Coordinate Solving. Evaluators should read the evaluation-set design and limitations before comparing scores. For everyone else, remember this: before blaming the robot's model, check what it can see and when it can look again.
Sam: A useful question to take back to the lab.