Chenxi Xie, Yuhui Wu, Qiaosi Yi, Lei Zhang
5 min
Abstract
Existing image editing methods can be generally categorized into textual instruction-based and visual prompt-based ones. Textual instructions are semantically expressive, but are limited by the coarse granularity of spatial control of the editing results. In contrast, visual prompts such as drag and point can provide precise spatial guidance, but are limited by the inherent ambiguity in semantic intent. To unify the strength of textual and visual prompts, we present Text-Vision Co-Instructed Image Editing, which jointly models textual instructions as semantic intent and sparse visual instructions as spatial guidance, aiming to achieve precise and intent-faithful image manipulation. To this end, we first construct a textual-visual instruction paired dataset with more than 23K samples derived from dynamic videos, enabling aligned supervision for cross-modal instruction. We then propose TV-Edit, a Textual-Visual instruction unified Editing framework to contextualize drag or point-based visual instructions with image-text semantics and lift them into semantic-aware control representations for pretrained editing backbones. By integrating semantic intent and spatial constraints, TV-Edit leads to more precise spatial control, less instruction ambiguity, and stronger structural consistency than text-only or drag-based alternatives. Finally, we establish TV-Edit-Bench, a deliberately designed benchmark to evaluate semantic faithfulness, spatial alignment, and visual consistency with ground-truth references and controlled textual-visual variations for reliable assessment. Our experiments across multiple editing backbones demonstrate that TV-Edit consistently yields more precise and intent-faithful edits, significantly outperforming state-of-the-art instruction-based and drag-based baselines.
Alex: So they're not just asking "does it look nice?"—they're checking whether it actually did what the user intended.
Sam: Exactly. And to run those checks efficiently, they built a lightweight controller that sits alongside the main AI model. Think of the main model as a large engine generating the image, and the controller as a smaller unit handling only the spatial guidance. This keeps the system practical without requiring enormous computing power.
Alex: But if the controller is that small, can it actually influence the output in a meaningful way?
Sam: That's a fair concern. The solution is something they call time-modulated injection. Think of it like painting—you work differently at different stages. Early on, you're blocking in broad shapes. Later, you're adding fine detail. The controller's influence is strongest in the early stages, when the AI is still deciding on structure. By the time it's refining textures, the broad movements are already locked in.
Alex: Like a dimmer switch—turned up high when the AI is sketching shapes, turned down when it's finishing the surface detail.
Sam: That's a good way to put it. And to avoid disrupting the main model's existing knowledge, they only train the controller and the point encoder—the part that reads the drag points—from scratch. The main model stays frozen. So the creative capability is preserved, while the spatial reasoning is learned on top of it.
Alex: Training the spatial brain while leaving the creative brain untouched.
Sam: Precisely. It's a careful division of labour, and it's what allows the system to be both flexible and predictable.
Alex: It sounds like a meaningful step forward. Though I imagine it's not without limitations.
Sam: The authors are clear about that. Because it relies on large foundation models, it isn't fast enough for real-time interaction yet. It also handles 2D movements well, but struggles with complex 3D rotations—cases where an object needs to turn fully in space, not just shift position.
Alex: So it's a capable tool for designers who need precision, but instant, 3D-aware editing is still some way off.
Sam: That's a fair summary. The paper presents it as a foundation—a meaningful improvement in how intent and action are connected in AI image editing, and a base that future work can build on to address those remaining challenges. Thanks for listening to ResearchPod.