Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation

Sam: A robot that can do every step can still botch the whole job. It repeats a step, skips one, or picks up the wrong thing. This work argues that a planning layer outside the robot's action model can fix much of that. The layer tracks which step is active, where the relevant objects are, and when a step is finished.

Alex: Before the details, tell me where this comes from. Whose work is it, and what kind of paper is it?

Sam: It's an arXiv preprint from October 2026. The team is from the Chinese University of Hong Kong, Xiaomi, and the National University of Defense Technology. It's a method paper that also introduces a new benchmark. The system is called Orchestra.

Alex: I don't work in manipulation. Why should someone like me care?

Sam: Most recent progress has come from vision-language-action models. Those are networks that take camera images and an instruction and output robot motions. They're good at short, local actions. Long instructions with many steps are still hard. This paper says you don't have to cram that into the policy. You can coordinate it from outside. So we'll cover the idea, the evidence, and where it falls short.

Alex: Start with what was already tried. Surely people have given these policies memory.

Sam: They have. One earlier system, Mem-zero, came with a benchmark called RMBench, built for tasks that need memory. Its approach puts explicit memory components into the policy. The authors' point is that memory alone doesn't keep the robot's visual history and its task record lined up with the step it's actually on. There's a second problem too. A grasp can work mechanically and still target the wrong object.

Alex: And there are agent-style systems as well, where a language model drives the robot.

Sam: Right, and the closest one is HarnessVLA. It wraps a frozen action model and combines it with hand-built manipulation routines, steered by a memory-guided agent. Orchestra keeps that agent idea. It adds something the authors call a shared execution context, and a separate model that judges progress.

Alex: Walk me through it with a concrete instruction.

Sam: Take one of theirs: "Arrange the nameplates on the blue mat in the order three, one, two." First, a language-model agent splits that into ordered subtasks. Place nameplate three, then one, then two. That's the planning half, what the title calls letting the mind conceive. For the active subtask, a pointing model finds the relevant nameplate and its destination in the head camera. A segmentation model then outlines them and tracks them as things move. That's spatial grounding, meaning the system marks which image region is the intended object.

Alex: So the action model gets the current subtask plus a picture with the right object outlined.

Sam: Plus the recent camera frames and the robot's joint state. The action model is an adapted version of pi zero point five, a pretrained robot policy. They added two things to it. The object outlines and coordinates go into its input. They also gave it a short visual memory of the last few frames. The memory is mixed in early in the image encoder, so the language model sees no extra tokens.

Alex: Here's my worry. How does the system know a subtask is actually done? That seems like where it would fall apart.

Sam: That's the second key piece. A separately trained progress evaluator looks at the same context and predicts how far along the subtask is. It sorts progress into eleven categories. When its top guess is the final category, the harness records the step, drops the pending actions, and moves on. It also wipes the short-term visual memory and the grounding, but keeps the record of completed steps. The training labels are automatic, taken from how far through each demonstration clip a frame sits.

Alex: So the policy only ever has to think about one short step. The bookkeeping lives outside it.

Sam: That's the bet. Which brings us to the new benchmark that tests it, RMBench-Compose. They adapted five RMBench environments, things like pressing buttons, covering and uncovering blocks, and placing blocks on mats. The policy is trained only on single atomic skills and short demonstrations. At test time it gets one full instruction and has to chain six to eighteen subtasks.

Alex: How do they score that?

Sam: The main score is Average Success Length. It counts how many subtasks the robot completes in the requested order before the first failure. Orchestra averaged about seven point four subtasks. HarnessVLA averaged about three, so Orchestra goes roughly two and a half times as far.

Alex: Is that the harness doing the work, or the upgraded policy?

Sam: Both, and the paper is candid about it. With the plain pi zero point five policy inside, Orchestra roughly tripled the plain policy's success length. But it still scored below HarnessVLA. Only the full system, with harness plus adapted policy, pulls clearly ahead. They also ran removal experiments. Taking out the task planning and just feeding the whole instruction to the policy dropped the average from about seven point four to one and a half. Removing the short visual memory hurt badly too. Removing grounding hurt less, but a lot on the covering task.

Alex: And the progress evaluator, the piece I was worried about?

Sam: They swapped it for the simulator's own ground-truth signal of when a step is done. The scores were almost identical. The authors are careful here. They say rollout performance was similar, not that the evaluator detects completion equally well.

Alex: What about the original memory benchmark, where Mem-zero lives?

Sam: Across nine RMBench tasks, Orchestra averaged about sixty-eight percent success, against forty-two for Mem-zero. Each task had a hundred rollouts. One caveat on that comparison. Most baseline numbers are the officially reported ones, while HarnessVLA and Orchestra were run by the authors. And it's uneven. On one task, Swap T, Orchestra succeeded only ten percent of the time, and on another it trailed Mem-zero.

Alex: Simulation is forgiving. Did any of this survive a real robot?

Sam: They tested a dual-arm robot on four tasks, like breakfast preparation and sorting objects into a trash bin. Each had three to five subtasks. They ran twenty instructions per task for each configuration. Average Success Length rose by twenty-five percent. But notice what the comparison was. It was full Orchestra against Orchestra with the base policy. There's no outside baseline on real hardware.

Alex: That matters for the simulation numbers too. Were the outlines there predicted, or handed over?

Sam: Handed over. In simulation, the segmentation masks come from the environment's ground truth. Only the real-world runs use predicted pointing and segmentation. The authors name this themselves. They say their next step is longer real-world sequences with predicted grounding.

Alex: When it fails on the real robot, where does it fail?

Sam: They looked at sixty-five real-world failures. Close to nine in ten were plain action execution. The robot misplaced a fork, grasped a bottle wrongly, knocked an apple away, or toppled a number plate. A small minority were late completion detection, wrong objects, or a planning slip. In one case the agent listed a snack bag twice and forgot a box entirely.

Alex: So the coordination layer seems to be doing its job, and the hands are now the bottleneck.

Sam: Within that evaluation, yes, that's how the authors read it. They also say the examples argue for checking plans, grounding, and completion more carefully.

Alex: So who should sit down with the full paper, and where should they start?

Sam: Anyone building long-horizon systems around pretrained robot policies. Start with the method section on progress estimation and closed-loop execution. That's where the transition logic lives. Then read the ablations, and the failure-case analysis in the appendix. If you design benchmarks, look at how RMBench-Compose separates training on single skills from testing on long chains.

Alex: And for everyone else?

Sam: A robot needs to know when a step is finished, not just how to do it. Keep that judgment outside the hands, and the hands can go further.

Alex: Good one to keep in mind next time a demo looks flawless.