Marco Moletta, Michael C. Welle, Danica Kragic
8 min
Abstract
Humans naturally develop preferences for how manipulation tasks should be performed, which are often subtle, personal, and difficult to articulate. Although it is important for robots to account for these preferences to increase personalization and user satisfaction, they remain largely underexplored in robotic manipulation, particularly in the context of deformable objects like garments and fabrics. In this work, we study how to adapt pretrained visuomotor diffusion policies to reflect preferred behaviors using limited demonstrations. We introduce RKO, a novel preference-alignment method that combines the benefits of two recent frameworks: RPO and KTO. We evaluate RKO against common preference learning frameworks, including these two, as well as a baseline vanilla diffusion policy, on real-world cloth-folding tasks spanning multiple garments and preference settings. We show that preference-aligned policies (particularly RKO) achieve superior performance and sample efficiency compared to standard diffusion policy fine-tuning. These results highlight the importance and feasibility of structured preference learning for scaling personalized robot behavior in complex deformable object manipulation tasks.
Sam: One common way, called DPO, looks at how much better the winner's predicted actions score over the loser's, using the base model as a neutral yardstick—like judging a race by how far ahead the favorite pulls from the also-ran. Another, RPO, goes further by measuring how similar a pair looks in the robot's view; close calls get extra weight in training, since those are where the robot most needs to learn fine differences. A third, KTO, simplifies to straight binary labels per example—win or lose—without pairing, like sorting mail into keep or toss piles.
Alex: So RPO zooms in on tricky pairs that look alike, while KTO just flags each one independently. How does RKO pull the best from both?
Sam: RKO fuses KTO's simple win/lose tags with RPO's similarity check. It first labels everything binary, then reweights training emphasis using a distance measure between similar states—closer ones amplify the push toward winners. Picture a coach not just marking wins and losses, but yelling extra loud during drills for barely-won plays, honing the edge where errors lurk.
Alex: That logic tracks—focusing effort where preferences are subtle makes sense for floppy shirts.
Alex: Right, but to get why RKO sharpens those subtle edges so well, walk me through how it tweaks the robot's action planner during training—like, what's happening step by step inside?
Sam: These robot planners work by starting with random noise for actions—like scrambling a blurry picture of arm movements—and gradually sharpening it into clear steps, using camera views as a guide. Each sharpening pass predicts and subtracts noise, repeating over many rounds until the actions look right for folding. They call this a denoising diffusion process, but it's basically turning chaos into precise picks and places for the cloth.
Alex: So the robot imagines noisy action sketches from visuals, then cleans them up iteratively. How do the preference methods—like RKO—fit into that cleanup?
Sam: In training, they add noise to good and bad example actions, then compare how well the robot predicts that noise versus a frozen starting version. For RPO, it checks every good example against all bad ones in a group, giving extra push to pairs that look alike from the camera—using a closeness score from image features, like spotting twins in a crowd. KTO skips pairs, just tags each as win or lose and adjusts based on how much better or worse the prediction pulls away from the start.
Alex: What makes RKO's mix click for cloth, exactly?
Sam: RKO tags everything binary like KTO—no pairs needed—then layers on RPO's group weighing: for each good sample, it measures closeness to every bad one via cosine distance on visual embeddings, amping up the training signal where they're semantically near. This focuses cleanup on fuzzy spots where neat folds barely beat messy ones, like drilling free throws after close games.
Alex: Huh, so by blending tags and closeness, it hones the noise prediction right where styles blur. That seems like a smart way to personalize without overload.
Sam: Yes, and for deformable cloth with its pick-place twists, this precision matters. It points to preference tweaks scaling well beyond shirts.
Alex: That weighting on close calls sounds key for cloth's messiness. How did they set up the actual data to test this on real shirts and pants?
Sam: They used 60 good examples per preference style and 60 bad ones from other styles, split as 40 shared basics plus 20 fresh ones where a person jumps in mid-task to demo the preferred fix—called human takeover, since it grabs rare cloth flops the robot might hit. Three styles cycle roles: one supplies goods, another bads, the third trains a neutral starting point. All from real robot runs on trousers, sleeves, and t-shirts.
Alex: Human takeover fills gaps in weird flops, cycling keeps it fair. So with that data, how did RKO stack up against the others in practice?
Sam: On real dual-arm robots with wrist and overhead cameras, preference methods beat plain copying overall, with RKO topping four of nine garment-style combos. The paper suggests its edge comes from that border focus, adapting styles a clear step better under data limits. Training also wraps up faster.
Alex: Makes sense—the weights sharpen just where cloth tricks matter most.
Sam: Yes, though it has limits—the method needs balanced sets of winning and losing examples to shine, and dips in very low-data cases like 20 demos, where the robot hits unfamiliar situations not covered in training. Without the reweighting part, scores drop across tests, confirming that piece adds value. Failures mostly stem from those unfamiliar states, more common in two-arm tasks.
Alex: Right, so it thrives with decent data balance but flags on extremes or surprises. Still, for household robots, that points to picking up your folding quirks from a handful of shows.
Sam: Exactly. The work suggests preference tweaks like this offer a practical way to personalize robot handling of floppy objects without full retrains, using limited user demos alongside alternatives as losers. It builds on pretrained skills while steering clear of disliked paths, which boosts both accuracy and speed.
Alex: Well put, Sam. Thanks for breaking it down so clearly. That's it for this look at preference-aligned robot folding on ResearchPod.