Modular Diffusion Models for Structured Visual Recognition

Traditional computer vision models for tasks like object detection, instance segmentation, and scene graph generation typically produce deterministic, single-point estimates. This approach fails to capture the inherent uncertainty and ambiguity found in complex visual scenes, such as occluded objects or ambiguous boundaries. The authors seek to address this by introducing a generative framework that learns a full probability distribution over structured outputs.

To overcome the challenges of modeling heterogeneous and interdependent data, the authors propose Modular Diffusion Models (MDMs). Instead of training a single, monolithic diffusion model to handle all output types simultaneously—which often leads to incoherent intermediate states and alignment issues—the MDM framework decomposes the process. It treats the structured output as a collection of task-specific modules (e.g., one for spatial locations, one for categories, one for relationships). Each module is a separate diffusion process that can be trained independently. During inference, these modules are integrated seamlessly to generate a complete, coherent structured representation of the scene.

MDMs demonstrate significant improvements over existing diffusion-based methods. In object detection and instance segmentation, the framework achieves gains of up to 8.4 AP_mask and 3.9 AP_box compared to previous diffusion-based approaches, while matching the performance of strong deterministic models. Furthermore, in scene graph generation, MDMs outperform competitive baselines by 2.6 on hR@50/100. Beyond raw metrics, the model's ability to sample from the learned distribution allows it to provide multiple plausible interpretations for a single image, effectively quantifying prediction uncertainty.

This work provides a scalable and flexible way to apply diffusion models to structured prediction. By avoiding the complexity of joint training, the modular design makes it easier to handle diverse data types (continuous vs. categorical) and allows for fine-grained control over the generation process. This is particularly valuable for applications in autonomous driving, medical imaging, and robotics, where understanding the uncertainty of a prediction is as critical as the prediction itself.