Yue Gong, Le Zhang, Rui Chen, Song Chang, Xiaochao Qu, Luoqi Liu, Ting Liu, Yebin Liu
8 min
Abstract
We present PhysInsert, a framework for physically plausible reference-based object insertion that jointly adapts objects and their surrounding scenes. Existing methods often rely on masked inputs, generation-based pairs, or training pairs with limited viewpoint variation, leaving pose and illumination adaptation and bidirectional object-scene interactions insufficiently constrained. We construct the Insert World dataset with strictly aligned Blender-rendered counterfactual pairs and object references across viewpoints and lighting conditions. Building on these data, Geometry-Guided Adaptation couples a VGGT geometry stream with image and text streams under target-scene feature supervision to guide spatially consistent insertion. Complementary 2D Object-Scene Spatial Supervision jointly supervises object occupancy and environmental changes while grounding spatial instructions in object regions through image-text attention. We introduce PhysInsert-Bench with dedicated cross-environment and cross-view cases requiring illumination and pose adaptation beyond appearance copying. Experiments on PhysInsert-Bench and DreamBooth demonstrate improved image fidelity; a user study further shows more accurate placement and improved lighting integration, perspective, and pose plausibility.
Sam: That is their stated motivation. The renderer recomputes shadows, mirror images, and reflections after removal. So the pair captures not just the object's pixels, but the changes its presence causes elsewhere.
Alex: What makes those pairs useful for adaptation, rather than simply teaching the model to put back what disappeared?
Sam: They separately render each object across lighting environments and viewpoints. The reference can therefore differ from the target in both respects. Their InsertWorld training set contains about thirteen thousand aligned scene pairs, alongside a much larger collection of object references.
Alex: Better examples could explain a lot by themselves. What is the extra idea in the model?
Sam: They add a geometry stream: evolving internal features that represent scene structure. It begins with features extracted from the empty background by a frozen geometry model called VGGT. That model predicts camera information, depth, and scene structure.
Alex: But the empty background cannot tell you the geometry of an object that is not there. How do they bridge that gap?
Sam: During training, they also extract geometric features from the completed target scene. The evolving geometry stream learns to predict those features while exchanging information with image and text streams. The completed target provides supervision, but its features are never fed into the generator.
Alex: So at generation time, it starts from the background and works toward a plausible completed structure. It does not receive the answer.
Sam: The geometry stream remains active at inference, but the target image is unavailable. That distinction matters. The authors train an intermediate representation of the completed scene, rather than merely attaching unchanged background geometry to the editor.
Alex: Geometry alone does not determine a shadow or reflection. How do they teach those changes?
Sam: They add image-space supervision for both object occupancy and external effects. Outside the object region, differences between the aligned renders become a soft map of visible environmental changes. That map is a proxy, not a separate physical explanation of every shadow or reflection.
Alex: And there is still the placement instruction. How does “in front of the plate” become a location rather than just more text?
Sam: They use spatial grounding, meaning a connection between placement language and image regions. During training, attention to spatial words is encouraged to align with the object's region. The complete background stays available, unlike approaches that remove information inside an editing mask.
Alex: Give me one example where these pieces have to work together.
Sam: In the beverage-can comparison, the model adapts the can to the scene's lighting and produces its reflection in adjacent glass. Copying the reference appearance would not solve both demands. The object must change, and something outside its boundary must change too.
Alex: Let's turn to evidence. Does their evaluation actually require those adaptations, rather than rewarding attractive copies?
Sam: Their controlled benchmark contains four hundred cases from scenes absent from training. It separates optical challenges from structural ones. Cases test lighting changes, environmental effects, and pose adaptation using references viewed from different angles.
Alex: What is the clearest result to hold onto?
Sam: With placement specified through text, their placement score is about sixty-three. Nano Banana Pro, the strongest comparison on that measure, scores about fifty-five. The measure combines overlap with a penalty for displaced object centers; it is not a percentage of successful edits.
Alex: That is useful, but their own dataset and benchmark favor the problem they chose. Can we separate the method from simply getting more relevant training?
Sam: They fine-tune existing insertion models on InsertWorld with matched training budgets, and those models improve. They also compare their full system against ordinary dataset fine-tuning. Removing the geometry, spatial grounding, or object-effect supervision worsens every reported metric in the component tests.
Alex: That supports contributions from both data and architecture. How far does the result travel beyond rendered scenes?
Sam: They test DreamBooth objects and fifty in-the-wild cases. PhysInsert leads the reported full-image fidelity measures in those settings. But it does not beat every compared model on object identity in the wild.
Alex: And physical plausibility is harder to measure than image similarity. What evidence comes from people?
Sam: A blinded study has twenty-seven participants rating separate real-image insertion cases. PhysInsert receives the highest ratings for lighting integration, perspective, pose, and overall quality. It remains competitive, rather than uniformly best, on identity and background preservation.
Alex: What should keep us from treating that as a general certificate of physical correctness?
Sam: Placement relies on automatic localization and segmentation, which can make mistakes. Depth consistency compares estimated depths, not measured geometry. And the language-model plausibility judge gives PhysInsert only a small mean advantage over Nano Banana Pro, with overlapping reported confidence intervals.
Alex: What is the limitation that matters most if I want to use it?
Sam: Identity under large viewpoint changes. The authors do not explicitly optimize identity preservation, and acknowledge that distinctive features can be lost. The added geometry stream also increases inference time and memory compared with the underlying editor.
Alex: Given those trade-offs, who should read the full document, and where should they start?
Sam: Researchers building reference-based editors should start with Dataset Construction and Geometry-Conditioned Object Insertion. Then read Ablation Studies to separate the value of the data from the added supervision. Read Limitations before assuming that better pose adaptation guarantees faithful identity.
Alex: And for researchers outside image editing, what is the one idea worth keeping?
Sam: A convincing insertion is not just an object that fits the scene. It is a scene that responds to the object.
Alex: Thanks for listening.