Instruction-based Image Editing: A Survey on Data, Models, Evaluation, and Applications

Instruction-based Image Editing (IIE) represents a paradigm shift in visual content creation, moving from manual, tool-heavy workflows to natural language-driven manipulation. This survey systematically organizes the field by defining a hierarchical taxonomy of editing tasks, ranging from simple atomic operations (like object removal or tone adjustment) to complex compositional tasks (such as sequential reasoning or spatial adjustments). By tracing the evolution from GAN-based architectures to modern diffusion and autoregressive models, the authors provide a clear roadmap of the technological milestones that have enabled current one-sentence editing capabilities.

A significant portion of the paper is dedicated to how training data is constructed for these models. The authors highlight a transition from implicit, generation-based pipelines to modular, structure-aware approaches. For instance, tasks like object removal and replacement now frequently leverage advanced segmentation models (e.g., SAM) and vision-language models (e.g., Qwen2.5-VL) to ensure that edits are semantically accurate and visually consistent with the surrounding scene. This modularity allows for more precise control and better alignment between textual instructions and visual outcomes.

To address the lack of standardized evaluation in the field, the authors introduce the CDD-IIE Bench. This benchmark is designed to be diagnostic, breaking down performance into 5 principal categories and 21 fine-grained sub-criteria. By empirically testing leading open-source solutions, the survey identifies critical gaps in how models handle complex instructions and spatial reasoning, providing researchers with a clearer understanding of where current state-of-the-art systems succeed and where they struggle.