DiFA: Inference-Time Forward-Process Alignment for Diffusion Models

Diffusion models typically rely on numerical solvers that treat each denoising step as an independent, locally linear estimation problem. This approach often ignores the statistical uncertainty inherent in the model's predictions and the temporal redundancy present across the sampling trajectory. The authors ask whether it is possible to improve generative fidelity by treating these sequential predictions as correlated observations of a stable underlying signal, rather than just isolated steps in a numerical integration process.

The authors propose Forward-Process Aligned Diffusion prediction (DiFA), a training-free framework that refines clean-signal predictions during inference. DiFA constructs a temporal consensus by aggregating historical predictions using weights derived from the forward diffusion process's noise levels. This is inspired by Kalman filtering, where observations are fused based on their reliability. To prevent the over-smoothing that often accompanies such averaging, the authors introduce a 'deviation guidance' mechanism that adaptively preserves high-frequency details by comparing the current prediction against the consensus anchor.

DiFA consistently improves generative performance across standard benchmarks, including CIFAR-10 and ImageNet, as measured by metrics like FID, IS, and FD-DINOv2. Because the framework operates by refining the output of the existing denoiser before the solver step, it achieves these gains without requiring any additional network evaluations (NFEs) or expensive retraining. The results demonstrate that aligning the inference process with the statistical structure of the forward diffusion process effectively mitigates the cumulative prediction drift that plagues standard few-step samplers.

This work provides a principled, training-free way to boost the performance of pre-trained diffusion models. By shifting the focus from purely numerical integration to statistical state estimation, DiFA offers a way to extract higher-quality samples from existing models, making it a highly practical tool for researchers looking to improve generation quality without the overhead of distillation or architectural changes.