Efficient Fairness Auditing Across Guidance Scales in Text-to-Image Diffusion Models via Causal Abstraction

Fairness auditing for text-to-image diffusion models is computationally expensive because it requires generating large numbers of images across various prompts, random seeds, and classifier-free guidance (CFG) scales. The authors ask whether a causal abstraction framework can create an efficient, high-level surrogate model that preserves the interventional behavior of the original diffusion process, allowing for faster fairness audits.

The researchers represent the diffusion process as a low-level structural causal model (SCM) and map it to a high-level model over abstract denoising states. They use a probabilistic transformer to learn the dynamics of these abstract states. By treating the CFG scale as an intervention variable, the model predicts how demographic attributes change without needing to perform the full, iterative denoising process for every sample. The authors establish theoretical conditions under which this lossy abstraction remains valid for specific fairness queries, even when the learned high-level causal graph differs from the projected low-level graph.

The proposed instrument achieves a 19.4x speedup in inference time compared to the original Stable Diffusion model. Experiments on standard Stable Diffusion v1.5 and the fairness-enhanced StayFair model demonstrate that the high-level surrogate accurately reproduces the prompt- and scale-dependent fairness trends observed in the full model. The instrument successfully captures the sensitivity of demographic representation to guidance interventions, providing a reliable way to detect fairness violations across a range of settings.

This work provides a principled, computationally efficient method for auditing generative AI systems. By moving from expensive, full-model simulations to amortized high-level causal models, researchers can perform more comprehensive fairness evaluations. This approach is particularly valuable for assessing how inference-time settings, such as guidance scales, impact model behavior, helping to ensure that fairness mitigations are robust across different usage scenarios.