Existing image editing methods can be generally categorized into textual instruction-based and visual prompt-based ones. Textual instructions are semantically expressive, but are limited by the coarse granularity of spatial control of the editing results. In contrast, visual prompts such as drag and point can provide precise spatial guidance, but are limited by the inherent ambiguity in semantic intent. To unify the strength of textual and visual prompts, we present Text-Vision Co-Instructed Image Editing, which jointly models textual instructions as semantic intent and sparse visual instructions as spatial guidance, aiming to achieve precise and intent-faithful image manipulation. To this end, we first construct a textual-visual instruction paired dataset with more than 23K samples derived from dynamic videos, enabling aligned supervision for cross-modal instruction. We then propose TV-Edit, a Textual-Visual instruction unified Editing framework to contextualize drag or point-based visual instructions with image-text semantics and lift them into semantic-aware control representations for pretrained editing backbones. By integrating semantic intent and spatial constraints, TV-Edit leads to more precise spatial control, less instruction ambiguity, and stronger structural consistency than text-only or drag-based alternatives. Finally, we establish TV-Edit-Bench, a deliberately designed benchmark to evaluate semantic faithfulness, spatial alignment, and visual consistency with ground-truth references and controlled textual-visual variations for reliable assessment. Our experiments across multiple editing backbones demonstrate that TV-Edit consistently yields more precise and intent-faithful edits, significantly outperforming state-of-the-art instruction-based and drag-based baselines.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new way to bridge the gap between how we tell AI to edit images and how it actually performs those edits.
Sam: The paper we're discussing is called "Text-Vision Co-Instructed Image Editing." The central puzzle is that current AI editing tools are split into two camps: those that listen to text instructions, and those that follow visual cues. Neither approach works well on its own.
Alex: So the question is: how do you get the best of both worlds—the descriptive power of language and the precision of visual guidance?
Sam: Think of it like a conductor and a musician. Text instructions are like a conductor describing the mood of a piece—expressive, but vague about exact timing. Visual prompts, like dragging a point on a screen, are like sheet music telling the musician exactly where to place their fingers. Only the mood, and the performance gets sloppy. Only the notes, and you miss the emotional intent. You need both.
Alex: So if I want to open a crocodile's mouth in a photo, the text tells the AI "open the jaw"—but the AI might not know exactly which part to move, or how far.
Sam: Precisely. A text-only model might change the texture of the skin, or distort the entire head, because it has no spatial map. A drag-only model might move the pixels you clicked on, but without understanding that the goal is to "open a mouth," it could just warp the jaw into an unnatural shape. This research proposes a framework called TV-Edit to address that.
Alex: And how does it actually combine two such different types of information?
Sam: They built a component called a Content-Aware Spatial Controller. Think of it as a translator sitting between the text and the visual points. Instead of feeding the AI a raw coordinate for where to drag, this controller looks at the image content, the text instruction, and the movement points all at once—and fuses them into a single, coherent set of instructions the AI can act on.
Alex: So it's not just "move these pixels here." It's "move these pixels here *because* the goal is to open the mouth."
Sam: That is the core mechanism. To teach the model how these two signals should work together, the researchers also built a dataset of 23,000 examples. They took real videos, tracked how objects moved frame by frame, and paired those movements with descriptions of what was happening. This gave the model a large library of intent paired with action.
Alex: That sounds like a substantial amount of groundwork. How do they measure whether it actually works better than what came before?
Sam: They created a benchmark—a standardised test—called TV-Edit-Bench. It contains 120 test cases, each with a known correct result. The benchmark measures three things: how well the AI follows the meaning of the text, how accurately it aligns with the visual drag points, and how consistent the final image looks overall.
Alex: So they're not just asking "does it look nice?"—they're checking whether it actually did what the user intended.
Sam: Exactly. And to run those checks efficiently, they built a lightweight controller that sits alongside the main AI model. Think of the main model as a large engine generating the image, and the controller as a smaller unit handling only the spatial guidance. This keeps the system practical without requiring enormous computing power.
Alex: But if the controller is that small, can it actually influence the output in a meaningful way?
Sam: That's a fair concern. The solution is something they call time-modulated injection. Think of it like painting—you work differently at different stages. Early on, you're blocking in broad shapes. Later, you're adding fine detail. The controller's influence is strongest in the early stages, when the AI is still deciding on structure. By the time it's refining textures, the broad movements are already locked in.
Alex: Like a dimmer switch—turned up high when the AI is sketching shapes, turned down when it's finishing the surface detail.
Sam: That's a good way to put it. And to avoid disrupting the main model's existing knowledge, they only train the controller and the point encoder—the part that reads the drag points—from scratch. The main model stays frozen. So the creative capability is preserved, while the spatial reasoning is learned on top of it.
Alex: Training the spatial brain while leaving the creative brain untouched.
Sam: Precisely. It's a careful division of labour, and it's what allows the system to be both flexible and predictable.
Alex: It sounds like a meaningful step forward. Though I imagine it's not without limitations.
Sam: The authors are clear about that. Because it relies on large foundation models, it isn't fast enough for real-time interaction yet. It also handles 2D movements well, but struggles with complex 3D rotations—cases where an object needs to turn fully in space, not just shift position.
Alex: So it's a capable tool for designers who need precision, but instant, 3D-aware editing is still some way off.
Sam: That's a fair summary. The paper presents it as a foundation—a meaningful improvement in how intent and action are connected in AI image editing, and a base that future work can build on to address those remaining challenges. Thanks for listening to ResearchPod.