Physically-guided Image Generation for Multi-Projection Mapping

Projection Mapping (PM) involves superimposing digital content onto real-world 3D objects. While modern text-to-image diffusion models have simplified content creation, they are typically trained in an idealized 2D space, leading to visual artifacts like geometric misalignment and color clipping when projected onto physical surfaces. ConPhyG (Controllable Physically-guided Generative framework) addresses this by bridging the gap between generative AI and the physical constraints of multi-projector systems.

The paper introduces two distinct paradigms for PM content generation:

• Cooperative Paradigm: This approach treats the physical object as a partner. It injects multi-dimensional physical priors—specifically per-pixel gamut, depth, and edge maps—directly into the diffusion denoising process. This ensures that the generated content is physically harmonious with the object's surface.
• Adversarial Paradigm: This approach treats the object's physical properties as interference. It generates content in an unconstrained space and subsequently uses bounded numerical optimization to perform radiometric compensation, effectively 'correcting' the projection to match the desired visual output.

ConPhyG provides an interactive interface that allows users to toggle between these modes based on the specific alignment of their creative intent and the physical environment. Key technical components include:

• In-Gamut Determination Network (IGDN): A lightweight network that classifies whether target colors fall within the reproducible gamut of the multi-projector setup, enabling gamut-aware generation.
• ControlNet Integration: Used to enforce structural conditioning, ensuring that generated textures align with the geometry and edges of the physical object.
• Multi-Projector Optimization: A Bounded-Variable Least Square (BVLS) algorithm calculates optimal input intensities for each projector, ensuring seamless coverage and brightness consistency across overlapping projection areas.