Humans naturally develop preferences for how manipulation tasks should be performed, which are often subtle, personal, and difficult to articulate. Although it is important for robots to account for these preferences to increase personalization and user satisfaction, they remain largely underexplored in robotic manipulation, particularly in the context of deformable objects like garments and fabrics. In this work, we study how to adapt pretrained visuomotor diffusion policies to reflect preferred behaviors using limited demonstrations. We introduce RKO, a novel preference-alignment method that combines the benefits of two recent frameworks: RPO and KTO. We evaluate RKO against common preference learning frameworks, including these two, as well as a baseline vanilla diffusion policy, on real-world cloth-folding tasks spanning multiple garments and preference settings. We show that preference-aligned policies (particularly RKO) achieve superior performance and sample efficiency compared to standard diffusion policy fine-tuning. These results highlight the importance and feasibility of structured preference learning for scaling personalized robot behavior in complex deformable object manipulation tasks.
Alex: Welcome to another episode of ResearchPod. Sam, robots are getting good at folding clothes these days, but they often do it in their own rigid way, ignoring how a person might prefer it done more neatly or quickly.
Sam: That's right, Alex. This paper, titled "Preference Aligned Visuomotor Diffusion Policies for Deformable Object Manipulation" by Marco Moletta and colleagues, tackles exactly that gap. The core idea is adapting robot brains—pretrained on lots of examples—to pick up a user's specific folding style using just a handful of demonstrations from that user.
Alex: So the paper is basically asking how to teach a robot your personal way of folding a shirt without needing thousands of practice runs from you?
Sam: Yes, precisely. Robots handle stiff objects okay, but flexible ones like shirts or pants bend and flop in endless ways, making every fold tricky to predict or copy perfectly. Users have subtle preferences—like folding sleeves flat or keeping edges crisp—that are hard to describe in words, so showing a few examples is more natural.
Alex: That makes sense for everyday stuff like laundry. But why is cloth folding such a tough test case here?
Sam: Cloth has complicated movements because it can twist, bunch up, or stretch differently each time, unlike a block that stays put. Pretrained robot systems use camera views to plan action steps, but they stick to average styles from big datasets. The challenge is shifting to your style—like neat creases versus quick flattens—with only 60 demos or so.
Alex: Okay, so it's not just about folding at all, but folding your way without tons of data. What's the key approach they propose for that?
Sam: They introduce RKO, which blends two smart ideas: one labels demos as good or bad per example, the other weighs similar-looking ones to focus training where choices are toughest. Like a coach spotting close games and drilling those plays hardest. This aligns the robot's actions to preferences with fewer examples than standard methods.
Alex: How do those building blocks actually work in practice for the robot's training?
Sam: First, they start with a base robot skill set trained on a big pile of neutral folding examples. Then, for your preferences, they gather a small set: winning demos of your neat style, and losing ones of messier alternatives. The training compares these pairs directly, pushing the robot to favor winners and avoid losers, which sharpens it faster than just copying winners alone.
Alex: Okay, so pairs of good and bad examples act like a tug-of-war to teach preferences. What are the specific ways they make that comparison?
Sam: One common way, called DPO, looks at how much better the winner's predicted actions score over the loser's, using the base model as a neutral yardstick—like judging a race by how far ahead the favorite pulls from the also-ran. Another, RPO, goes further by measuring how similar a pair looks in the robot's view; close calls get extra weight in training, since those are where the robot most needs to learn fine differences. A third, KTO, simplifies to straight binary labels per example—win or lose—without pairing, like sorting mail into keep or toss piles.
Alex: So RPO zooms in on tricky pairs that look alike, while KTO just flags each one independently. How does RKO pull the best from both?
Sam: RKO fuses KTO's simple win/lose tags with RPO's similarity check. It first labels everything binary, then reweights training emphasis using a distance measure between similar states—closer ones amplify the push toward winners. Picture a coach not just marking wins and losses, but yelling extra loud during drills for barely-won plays, honing the edge where errors lurk.
Alex: That logic tracks—focusing effort where preferences are subtle makes sense for floppy shirts.
Alex: Right, but to get why RKO sharpens those subtle edges so well, walk me through how it tweaks the robot's action planner during training—like, what's happening step by step inside?
Sam: These robot planners work by starting with random noise for actions—like scrambling a blurry picture of arm movements—and gradually sharpening it into clear steps, using camera views as a guide. Each sharpening pass predicts and subtracts noise, repeating over many rounds until the actions look right for folding. They call this a denoising diffusion process, but it's basically turning chaos into precise picks and places for the cloth.
Alex: So the robot imagines noisy action sketches from visuals, then cleans them up iteratively. How do the preference methods—like RKO—fit into that cleanup?
Sam: In training, they add noise to good and bad example actions, then compare how well the robot predicts that noise versus a frozen starting version. For RPO, it checks every good example against all bad ones in a group, giving extra push to pairs that look alike from the camera—using a closeness score from image features, like spotting twins in a crowd. KTO skips pairs, just tags each as win or lose and adjusts based on how much better or worse the prediction pulls away from the start.
Alex: What makes RKO's mix click for cloth, exactly?
Sam: RKO tags everything binary like KTO—no pairs needed—then layers on RPO's group weighing: for each good sample, it measures closeness to every bad one via cosine distance on visual embeddings, amping up the training signal where they're semantically near. This focuses cleanup on fuzzy spots where neat folds barely beat messy ones, like drilling free throws after close games.
Alex: Huh, so by blending tags and closeness, it hones the noise prediction right where styles blur. That seems like a smart way to personalize without overload.
Sam: Yes, and for deformable cloth with its pick-place twists, this precision matters. It points to preference tweaks scaling well beyond shirts.
Alex: That weighting on close calls sounds key for cloth's messiness. How did they set up the actual data to test this on real shirts and pants?
Sam: They used 60 good examples per preference style and 60 bad ones from other styles, split as 40 shared basics plus 20 fresh ones where a person jumps in mid-task to demo the preferred fix—called human takeover, since it grabs rare cloth flops the robot might hit. Three styles cycle roles: one supplies goods, another bads, the third trains a neutral starting point. All from real robot runs on trousers, sleeves, and t-shirts.
Alex: Human takeover fills gaps in weird flops, cycling keeps it fair. So with that data, how did RKO stack up against the others in practice?
Sam: On real dual-arm robots with wrist and overhead cameras, preference methods beat plain copying overall, with RKO topping four of nine garment-style combos. The paper suggests its edge comes from that border focus, adapting styles a clear step better under data limits. Training also wraps up faster.
Alex: Makes sense—the weights sharpen just where cloth tricks matter most.
Sam: Yes, though it has limits—the method needs balanced sets of winning and losing examples to shine, and dips in very low-data cases like 20 demos, where the robot hits unfamiliar situations not covered in training. Without the reweighting part, scores drop across tests, confirming that piece adds value. Failures mostly stem from those unfamiliar states, more common in two-arm tasks.
Alex: Right, so it thrives with decent data balance but flags on extremes or surprises. Still, for household robots, that points to picking up your folding quirks from a handful of shows.
Sam: Exactly. The work suggests preference tweaks like this offer a practical way to personalize robot handling of floppy objects without full retrains, using limited user demos alongside alternatives as losers. It builds on pretrained skills while steering clear of disliked paths, which boosts both accuracy and speed.
Alex: Well put, Sam. Thanks for breaking it down so clearly. That's it for this look at preference-aligned robot folding on ResearchPod.