Youpeng Wen, Gang Wang, Tianyu Huang, Weiwei Zhang, Zhijian Huang, Wei Chen, Xieyuanli Chen, Yunhui Liu
8 min
Abstract
A robot following a multi-step instruction needs to act on the right objects in the right order, even as the scene changes. It must also maintain awareness of task progress and determine when to advance. To address these challenges, we introduce Orchestra, an agent harness that separates task planning and execution management from action generation. Orchestra decomposes a complete multimodal instruction into ordered subtasks, tracks completed steps, and updates the locations of relevant objects and destinations during execution. It maintains a shared context containing the active subtask, spatial grounding, and recent observations. An action policy uses this context to generate actions, while a separately trained progress evaluator estimates when to advance to the next subtask. To evaluate whether robots trained on atomic instructions and short trajectories can execute longer subtask sequences, we introduce a benchmark named RMBench-Compose. Orchestra achieves a mean success rate of $68.22\%$ across nine RMBench tasks, outperforming Mem-0 by $26.22$ percentage points. On RMBench-Compose, it achieves an Average Success Length (ASL) of $7.35$ subtasks, $2.46$ times that of HarnessVLA. Across four real-world multi-step tasks, it improves mean ASL by $25.0\%$ over Orchestra using the base $π_{0.5}$ policy. These results show that Orchestra helps robots carry out longer sequences of subtasks while preserving the requested subtask order and acting on the specified objects. Our code and project page are available at https://youpengwen.github.io/Orchestra.
Alex: So the action model gets the current subtask plus a picture with the right object outlined.
Sam: Plus the recent camera frames and the robot's joint state. The action model is an adapted version of pi zero point five, a pretrained robot policy. They added two things to it. The object outlines and coordinates go into its input. They also gave it a short visual memory of the last few frames. The memory is mixed in early in the image encoder, so the language model sees no extra tokens.
Alex: Here's my worry. How does the system know a subtask is actually done? That seems like where it would fall apart.
Sam: That's the second key piece. A separately trained progress evaluator looks at the same context and predicts how far along the subtask is. It sorts progress into eleven categories. When its top guess is the final category, the harness records the step, drops the pending actions, and moves on. It also wipes the short-term visual memory and the grounding, but keeps the record of completed steps. The training labels are automatic, taken from how far through each demonstration clip a frame sits.
Alex: So the policy only ever has to think about one short step. The bookkeeping lives outside it.
Sam: That's the bet. Which brings us to the new benchmark that tests it, RMBench-Compose. They adapted five RMBench environments, things like pressing buttons, covering and uncovering blocks, and placing blocks on mats. The policy is trained only on single atomic skills and short demonstrations. At test time it gets one full instruction and has to chain six to eighteen subtasks.
Alex: How do they score that?
Sam: The main score is Average Success Length. It counts how many subtasks the robot completes in the requested order before the first failure. Orchestra averaged about seven point four subtasks. HarnessVLA averaged about three, so Orchestra goes roughly two and a half times as far.
Alex: Is that the harness doing the work, or the upgraded policy?
Sam: Both, and the paper is candid about it. With the plain pi zero point five policy inside, Orchestra roughly tripled the plain policy's success length. But it still scored below HarnessVLA. Only the full system, with harness plus adapted policy, pulls clearly ahead. They also ran removal experiments. Taking out the task planning and just feeding the whole instruction to the policy dropped the average from about seven point four to one and a half. Removing the short visual memory hurt badly too. Removing grounding hurt less, but a lot on the covering task.
Alex: And the progress evaluator, the piece I was worried about?
Sam: They swapped it for the simulator's own ground-truth signal of when a step is done. The scores were almost identical. The authors are careful here. They say rollout performance was similar, not that the evaluator detects completion equally well.
Alex: What about the original memory benchmark, where Mem-zero lives?
Sam: Across nine RMBench tasks, Orchestra averaged about sixty-eight percent success, against forty-two for Mem-zero. Each task had a hundred rollouts. One caveat on that comparison. Most baseline numbers are the officially reported ones, while HarnessVLA and Orchestra were run by the authors. And it's uneven. On one task, Swap T, Orchestra succeeded only ten percent of the time, and on another it trailed Mem-zero.
Alex: Simulation is forgiving. Did any of this survive a real robot?
Sam: They tested a dual-arm robot on four tasks, like breakfast preparation and sorting objects into a trash bin. Each had three to five subtasks. They ran twenty instructions per task for each configuration. Average Success Length rose by twenty-five percent. But notice what the comparison was. It was full Orchestra against Orchestra with the base policy. There's no outside baseline on real hardware.
Alex: That matters for the simulation numbers too. Were the outlines there predicted, or handed over?
Sam: Handed over. In simulation, the segmentation masks come from the environment's ground truth. Only the real-world runs use predicted pointing and segmentation. The authors name this themselves. They say their next step is longer real-world sequences with predicted grounding.
Alex: When it fails on the real robot, where does it fail?
Sam: They looked at sixty-five real-world failures. Close to nine in ten were plain action execution. The robot misplaced a fork, grasped a bottle wrongly, knocked an apple away, or toppled a number plate. A small minority were late completion detection, wrong objects, or a planning slip. In one case the agent listed a snack bag twice and forgot a box entirely.
Alex: So the coordination layer seems to be doing its job, and the hands are now the bottleneck.
Sam: Within that evaluation, yes, that's how the authors read it. They also say the examples argue for checking plans, grounding, and completion more carefully.
Alex: So who should sit down with the full paper, and where should they start?
Sam: Anyone building long-horizon systems around pretrained robot policies. Start with the method section on progress estimation and closed-loop execution. That's where the transition logic lives. Then read the ablations, and the failure-case analysis in the appendix. If you design benchmarks, look at how RMBench-Compose separates training on single skills from testing on long chains.
Alex: And for everyone else?
Sam: A robot needs to know when a step is finished, not just how to do it. Keep that judgment outside the hands, and the hands can go further.
Alex: Good one to keep in mind next time a demo looks flawless.