Yoad Tewel, Yuval Atzmon, Gal Chechik, Lior Wolf
6 min
Abstract
Modern generative models possess a deep understanding of visual content, yet training them for image editing typically requires massive datasets of paired examples. This limits scalability, especially for video editing where collecting paired data is prohibitively expensive. We propose Bootstrap Your Generator (ByG), a general framework for unpaired training of flow matching editing models. It leverages the base model's knowledge without any external signal. Our approach pairs instruction-following cues extracted from the frozen model with cycle-consistency for structure preservation. To make this tractable, we propose to route gradients from downstream losses over clean predictions to noisy training states. We demonstrate state-of-the-art results on challenging data-scarce image and video editing scenarios. Extensive evaluations and user studies show that our method effectively generalizes to unseen domains and outperforms supervised baselines trained on millions of samples. Analysis reveals that our gradient routing bridges the train-inference gap, and extracting semantic cues from a base model provides a robust training signal that obviates the need for external reward models.
Alex: So the invisible training wheels let the model learn from the messy intermediate steps without the whole thing falling apart?
Sam: Right. The specific technique is called "Straight-Through Estimation." Normally, the model can only learn from a clean, final image — but during training, it's working through many noisy, imperfect steps to get there. Straight-Through Estimation acts like a bridge: it lets the model look at the clean final result and work backwards to understand what it should have done at each messy step along the way.
Alex: So the model is essentially using its own output as a teaching signal. It's training itself.
Sam: That's the "bootstrapping" idea the name refers to. The model generates an edit, evaluates how good it was, and uses that evaluation to improve the next edit — without a human ever labeling what "correct" looks like. And the paper reports this approach outperforms models trained on millions of human-labeled pairs, which suggests that having more data isn't always the answer. Sometimes a smarter use of existing knowledge matters more.
Alex: Interesting. But here's something I'm still wondering about — how does the model know which part of the image to change, and which parts to leave completely alone?
Sam: This is where a technique the paper calls "Semantic Guided Directional Regularization" comes in. The word "semantic" here just means meaning — as in, the meaning of the text instruction. The model compares how it responds to the original description versus the new one, and calculates what you might call a "directional pull" — a specific direction of change in its internal representation. It then constrains itself to only move in that direction, leaving the rest of the image's structure untouched.
Alex: So it's isolating the difference between the two instructions and only applying that difference. That's why the background doesn't randomly shift when you only asked to change the texture of one object.
Sam: Exactly. And to prevent the edit from drifting too far or becoming unstable, the system also keeps a kind of anchor — a smoothed, stable version of its own weights that it checks itself against. Think of it like having a reference copy of your work that you're not allowed to stray too far from. It creates a constant tension between making the edit and preserving what was already there.
Alex: A tug-of-war between change and stability.
Sam: That's a fair description. And by managing that tension carefully, the system produces edits that users consistently prefer — even when it's applied to visual styles it has never encountered before, like voxel-style renderings, which look like everything is built from tiny blocks.
Alex: Are there tasks where this approach struggles?
Sam: Object removal is a clear weak point. Because the system relies heavily on text instructions, simply dropping a word from a description — saying "a dog" instead of "a dog wearing a hat" — doesn't give the model a strong enough signal to understand that something should be erased entirely. Removal is a fundamentally different kind of task from transformation, and the text-based approach doesn't handle it as cleanly.
Alex: So the framework is well-suited to changing things, but less so to making things disappear.
Sam: That's a fair summary of where the current limitations lie. What makes this work notable, though, is the broader implication: that we may not need to keep building larger and larger labeled datasets for every new editing task. If the knowledge is already inside the model, the challenge shifts to finding smarter ways to surface it.
Alex: That's a meaningful shift in how we think about training these systems. Thanks for walking through it, Sam — and thanks to everyone listening to ResearchPod.