Sinjae Kang, Dong Kyu Shin, Jaehong Kim, Kamin Lee, Hyoeun Kim, Kaixin Wang, Li Zhao, Seonghyeon Ye, Kimin Lee
7 min
Abstract
Robot policy evaluation often relies on a binary success signal for task completion. In real-world deployment, however, desired execution behaviors are also defined by quality attributes such as execution speed and motion smoothness, which need to be dynamically controlled according to task requirements. To address this, we introduce Quality-Controllable Policy (QCP), a robot policy that enables inference-time control over both individual quality attributes and their combinations. To extend the controllable quality range beyond initial demonstrations, we further propose an iterative hindsight relabeling loop: QCP collects online rollouts under target quality labels, adds successful trajectories to the dataset, relabels the accumulated trajectories in hindsight according to their measured quality, and fine-tunes the policy. Experiments on Robomimic and RoboEval demonstrate that QCP consistently improves both task execution quality and success rates across iterations, producing successful rollouts that exceed the quality of the initial demonstrations. Finally, we validate QCP on real-world manipulation tasks, where controlling speed increases throughput in conveyor operations and controlling transport stability improves success when carrying wine glasses, demonstrating that quality-controllable policies yield practical gains in physical deployment.
Sam: Now for the harder claim. If all those examples came from people, how does the robot become better than the examples?
Alex: The authors collect rollouts, meaning complete robot attempts, while requesting the best quality group. They retain only successful attempts and measure what each actually achieved. Then they recompute the quality groups across the entire accumulated dataset, including the original demonstrations.
Sam: The requested label and the achieved label can differ. A robot asked to go fast might finish slowly, so you train it under the speed it actually achieved.
Alex: That is the hindsight part. As better attempts accumulate, the boundary for the best group moves upward. The policy is trained again on those revised labels. The authors call this iterative hindsight relabeling.
Sam: That still needs some attempts to exceed the demonstrations in the first place. Where do those improvements come from?
Alex: The authors offer two explanations. The policy may recombine well-executed stages from different demonstrations, and its action generation is stochastic, so attempts vary. They attribute initial improvements to those mechanisms; the experiments do not separately establish each explanation.
Sam: Does “trained again” mean continuing from the previous policy? Moving the label boundaries sounds like it could create contradictory training targets.
Alex: They restart training each round. Small simulation policies start from random weights; the physical robot model starts from its pretrained checkpoint. Their reason is the one you identified: the old label-to-behavior mapping no longer matches the revised labels.
Sam: Let's turn to evidence. Could they get the same improvement simply by drawing more attempts from the original policy?
Alex: They tested that directly on a simulated smoothness task, with a budget of eighteen hundred attempts for each approach. One policy stayed frozen; the other collected attempts in batches and retrained between them. The frozen policy's successful attempts did not reach the final iterative policy's median smoothness.
Sam: That is a useful distinction: not just finding a rare good attempt, but changing the distribution of attempts. How broad were the other comparisons?
Alex: They tested manipulation tasks in Robomimic and RoboEval, averaging results across three random seeds. Comparisons included ordinary cloning on accumulated successes, quality-filtered cloning, style-learning methods, and reinforcement-learning methods given quality rewards. QCP generally improved quality and success together more effectively.
Sam: “More effectively” is not the same as winning every measure. Did other methods sometimes complete more tasks?
Alex: Yes. On the Square speed task, one reinforcement-learning baseline had higher final success, while QCP finished faster when successful. The defensible summary is a strong quality-and-success combination, not universal superiority on task completion.
Sam: What happened when they moved from simulation to hardware? The conveyor example sounds like a direct test of practical value.
Alex: On a fixed-speed conveyor, throughput rose from about eight cans per minute with ordinary cloning to thirteen with the final QCP policy. That outcome needs both successful grasps and fast enough execution. Quality-filtered cloning instead lost throughput over its training rounds.
Sam: And carrying glasses tests something different from speed. How much evidence did they have for learning stability?
Alex: The initial wine-serving dataset contained fifty successful demonstrations, selected from many more teleoperated attempts. The robot had to move the tray between tables without dropping a glass. QCP reduced average tray tilt relative to ordinary cloning.
Sam: Less tilt sounds useful, but only if the glasses actually arrive. Did success improve alongside that measurement?
Alex: Across training rounds, QCP's success rose from roughly thirty-one percent to forty-two percent. That is a useful gain, but most attempts still failed. And tilt was averaged only over successful episodes, so it should not be read as an all-attempt safety measure.
Sam: That leaves the definition of quality itself. Can the robot balance any qualities we might want, or only the ones the researchers made measurable?
Alex: Only the supplied measurements are established here. They demonstrated joint control and improvement across multiple attributes. But they selected attributes by task, excluding some that were unaffected or conflicted with the task. This is not evidence that arbitrary objectives can all improve together.
Sam: Who should read the full document, and where would you send them first?
Alex: Researchers working on robot imitation learning or deployment should start with “Iterative Hindsight Relabeling.” Then read the equal-budget frozen-policy comparison in the analysis, followed by the real-robot experiments. Together, those show the mechanism, its strongest check, and its practical scope.
Sam: And what should everyone else keep, without taking the method as a general solution to robot reliability?
Alex: You do not have to discard imperfect successes to learn better execution. Keep them, label how they succeeded, and let new successes raise the standard.
Sam: Thanks for listening.