Robot policy evaluation often relies on a binary success signal for task completion. In real-world deployment, however, desired execution behaviors are also defined by quality attributes such as execution speed and motion smoothness, which need to be dynamically controlled according to task requirements. To address this, we introduce Quality-Controllable Policy (QCP), a robot policy that enables inference-time control over both individual quality attributes and their combinations. To extend the controllable quality range beyond initial demonstrations, we further propose an iterative hindsight relabeling loop: QCP collects online rollouts under target quality labels, adds successful trajectories to the dataset, relabels the accumulated trajectories in hindsight according to their measured quality, and fine-tunes the policy. Experiments on Robomimic and RoboEval demonstrate that QCP consistently improves both task execution quality and success rates across iterations, producing successful rollouts that exceed the quality of the initial demonstrations. Finally, we validate QCP on real-world manipulation tasks, where controlling speed increases throughput in conveyor operations and controlling transport stability improves success when carrying wine glasses, demonstrating that quality-controllable policies yield practical gains in physical deployment.
Alex: A robot can finish the job and still be too slow or too unsteady. This study gives robots an adjustable quality setting, then improves what their best setting can achieve.
Sam: So the claim is not just better task completion. It is choosing how the robot completes the task, without being stuck at the quality of its demonstrations?
Alex: Yes. That is the proposal in a method preprint from Sinjae Kang and colleagues, with simulation and physical robot experiments. They call it Quality-Controllable Policy, or QCP. In their tests, repeated training improves execution quality and task success together.
Sam: That matters if you build robots for an actual workflow. A conveyor needs throughput; a tray of wine glasses needs stability. Let's unpack how the quality setting works, what improves with practice, and where the evidence stops.
Alex: Start with behavior cloning: learning actions by imitating demonstrations. The authors argue that ordinary cloning treats every successful demonstration as equally desirable. It can mix fast and slow executions, or smooth and jerky ones, without giving the user a way to choose.
Sam: But choosing an execution style already has a literature. What changes here, rather than just adding another label to the input?
Alex: Earlier quality-controllable policies learned from static datasets, so their training covered a bounded range of quality. This work adds a loop that collects new attempts and updates what the labels mean. The authors connect that loop to Quark, a related approach in text generation.
Sam: Then let's separate the adjustable policy from the improvement loop. How does the robot learn what a quality setting means?
Alex: Each successful demonstration gets a score from a rule-based measurement. For speed, a shorter completed trajectory gets a better score. The authors divide the demonstrations into four quality groups, each containing roughly the same share of the data.
Sam: So “best” means the best group in the current dataset, not an absolute promise about speed. What does the robot actually receive during training?
Alex: It receives recent observations and the demonstration's quality label, and learns to predict a short sequence of actions. Every successful demonstration contributes. In the physical experiments, the label goes into the model's language prompt, without changing its architecture.
Sam: Why not just keep the fastest demonstrations? That would also tell the robot which behavior you prefer.
Alex: Filtering throws away successful examples that may help teach task completion. QCP keeps them, but distinguishes their execution quality. The authors' argument is that slower successes still teach useful task behavior, while the requested label selects the preferred style.
Sam: Now for the harder claim. If all those examples came from people, how does the robot become better than the examples?
Alex: The authors collect rollouts, meaning complete robot attempts, while requesting the best quality group. They retain only successful attempts and measure what each actually achieved. Then they recompute the quality groups across the entire accumulated dataset, including the original demonstrations.
Sam: The requested label and the achieved label can differ. A robot asked to go fast might finish slowly, so you train it under the speed it actually achieved.
Alex: That is the hindsight part. As better attempts accumulate, the boundary for the best group moves upward. The policy is trained again on those revised labels. The authors call this iterative hindsight relabeling.
Sam: That still needs some attempts to exceed the demonstrations in the first place. Where do those improvements come from?
Alex: The authors offer two explanations. The policy may recombine well-executed stages from different demonstrations, and its action generation is stochastic, so attempts vary. They attribute initial improvements to those mechanisms; the experiments do not separately establish each explanation.
Sam: Does “trained again” mean continuing from the previous policy? Moving the label boundaries sounds like it could create contradictory training targets.
Alex: They restart training each round. Small simulation policies start from random weights; the physical robot model starts from its pretrained checkpoint. Their reason is the one you identified: the old label-to-behavior mapping no longer matches the revised labels.
Sam: Let's turn to evidence. Could they get the same improvement simply by drawing more attempts from the original policy?
Alex: They tested that directly on a simulated smoothness task, with a budget of eighteen hundred attempts for each approach. One policy stayed frozen; the other collected attempts in batches and retrained between them. The frozen policy's successful attempts did not reach the final iterative policy's median smoothness.
Sam: That is a useful distinction: not just finding a rare good attempt, but changing the distribution of attempts. How broad were the other comparisons?
Alex: They tested manipulation tasks in Robomimic and RoboEval, averaging results across three random seeds. Comparisons included ordinary cloning on accumulated successes, quality-filtered cloning, style-learning methods, and reinforcement-learning methods given quality rewards. QCP generally improved quality and success together more effectively.
Sam: “More effectively” is not the same as winning every measure. Did other methods sometimes complete more tasks?
Alex: Yes. On the Square speed task, one reinforcement-learning baseline had higher final success, while QCP finished faster when successful. The defensible summary is a strong quality-and-success combination, not universal superiority on task completion.
Sam: What happened when they moved from simulation to hardware? The conveyor example sounds like a direct test of practical value.
Alex: On a fixed-speed conveyor, throughput rose from about eight cans per minute with ordinary cloning to thirteen with the final QCP policy. That outcome needs both successful grasps and fast enough execution. Quality-filtered cloning instead lost throughput over its training rounds.
Sam: And carrying glasses tests something different from speed. How much evidence did they have for learning stability?
Alex: The initial wine-serving dataset contained fifty successful demonstrations, selected from many more teleoperated attempts. The robot had to move the tray between tables without dropping a glass. QCP reduced average tray tilt relative to ordinary cloning.
Sam: Less tilt sounds useful, but only if the glasses actually arrive. Did success improve alongside that measurement?
Alex: Across training rounds, QCP's success rose from roughly thirty-one percent to forty-two percent. That is a useful gain, but most attempts still failed. And tilt was averaged only over successful episodes, so it should not be read as an all-attempt safety measure.
Sam: That leaves the definition of quality itself. Can the robot balance any qualities we might want, or only the ones the researchers made measurable?
Alex: Only the supplied measurements are established here. They demonstrated joint control and improvement across multiple attributes. But they selected attributes by task, excluding some that were unaffected or conflicted with the task. This is not evidence that arbitrary objectives can all improve together.
Sam: Who should read the full document, and where would you send them first?
Alex: Researchers working on robot imitation learning or deployment should start with “Iterative Hindsight Relabeling.” Then read the equal-budget frozen-policy comparison in the analysis, followed by the real-robot experiments. Together, those show the mechanism, its strongest check, and its practical scope.
Sam: And what should everyone else keep, without taking the method as a general solution to robot reliability?
Alex: You do not have to discard imperfect successes to learn better execution. Keep them, label how they succeeded, and let new successes raise the standard.
Sam: Thanks for listening.