BoxCtrl: 3D-Aware Visual Prompting for Geometric Image Editing

Research Approach

BoxCtrl addresses the limitations of text-based and 2D-guided image editing by introducing a 3D-aware visual prompting framework. Instead of relying on ambiguous text or fragile 2D warping, the authors use RGB-colored 3D bounding boxes. By painting the three orthogonal faces of a bounding box with distinct colors, the model can explicitly encode an object's position, scale, and orientation. This visual prompt acts as a compact, intuitive instruction that allows the diffusion model to understand the geometric transformation required without needing complex 3D mesh reconstruction.

Training Strategy

The authors employ a two-stage training paradigm to ensure both foundational capability and real-world generalization. First, they perform Supervised Fine-Tuning (SFT) using a large-scale synthetic dataset created with the Kubric engine, which provides ground-truth geometric transformations. To bridge the domain gap between synthetic data and real-world images, they implement an online Reinforcement Learning (RL) stage. This stage uses a joint reward function—incorporating spatial alignment, rotation accuracy, and visual fidelity—to optimize the model's performance on unpaired real-world data.

Geometric image editing is a critical task for creative workflows, yet it remains difficult for standard diffusion models to execute precisely. By decoupling geometric control from appearance, BoxCtrl provides a robust, end-to-end solution that avoids the labor-intensive requirements of 3D asset manipulation or mesh recovery. The use of RGB-coded bounding boxes offers a standardized, user-friendly interface for non-experts to perform complex spatial edits while maintaining high visual fidelity and object identity.