Seungyong Lee, Hyun Jun Jang, Sangoh Kim, Sungjoon Park
4 min
Abstract
Virtual try-on (VTO) has made significant progress in realistically transferring garments onto a target person. Yet most systems give the user little control over how a garment should be worn -- its size (loose or fitted), style (e.g., tucked in or untucked, open or closed), and spatial placement on the body. We address this gap with two complementary contributions. First, we define and solve Visual-Instance-Prompt Segmentation via VIP-SAM: given a flatlay image of a garment, segment that specific instance in a photograph of a person wearing it. This is an instance-level task, distinct from the typically studied category-level segmentation. Second, we introduce CtrlVTON, a controllable VTO framework that recasts try-on as an image editing problem and adds segmentation masks as pixel-level control over garment layout, including style, size, and spatial placement on the body. VIP-SAM and CtrlVTON each achieve state-of-the-art results on their respective tasks. In particular, CtrlVTON generates images that follow user-provided layouts far more faithfully than the strongest proprietary editing systems while matching them on garment fidelity.
Sam: That's where another part of the system comes in. They use what the paper calls task-token adherence. The idea is straightforward: rather than leaving the model to infer what the user wants, you give it an explicit instruction—a short digital label, like "add this item" or "swap this item." It's a bit like the difference between asking someone to "do something with this jacket" versus handing them a note that says "put this jacket on top of the sweater, don't remove the sweater." The token removes the ambiguity.
Alex: So the model isn't just looking at the image—it's also reading a small set of rules about how the clothes should interact?
Sam: Precisely. The mask handles the where, and the task token handles the how. Together, they give the system a much clearer brief than it would otherwise have.
Alex: And does this actually produce better results than existing approaches?
Sam: The evidence suggests a meaningful improvement, particularly on the harder tasks. Layering and tucking—the kinds of styling decisions that tend to trip up older systems—are handled more reliably. The most common failure modes in earlier work were things like the AI generating extra fabric that wasn't there, or placing a garment in slightly the wrong position. The mask-based approach largely prevents those errors, because the model has a hard boundary to work within.
Alex: There's an important caveat there, though, isn't there? The system is only as good as the mask it's given.
Sam: That's right, and the paper is clear about it. If the mask the user provides is imprecise—if the stencil is slightly off—the output will reflect that. The quality of the control signal sets a ceiling on the quality of the result. It's a meaningful constraint, and one worth keeping in mind when evaluating what the system can actually do in practice.
Alex: So taken together, what does this represent for the field?
Sam: It's a shift in how the problem is framed. Earlier systems asked the AI to make all the decisions. CtrlVTON asks the AI to follow instructions. That might sound like a small change, but it moves the technology from something that produces a single, uncontrollable output to something that behaves more like an interactive editing tool. For anyone who's ever bought clothes online and been disappointed when they arrived, that's a direction worth watching.
Alex: Thanks for listening to ResearchPod.