Learning When to Denoise: Optimizing Asynchronous Schedules for Latent Diffusion

Multi-representation diffusion models improve visual synthesis by denoising multiple, complementary views of an image simultaneously. However, these models typically rely on a synchronous schedule where all representations are denoised at the same rate. This paper investigates whether learning an asynchronous schedule—where different representations evolve at different speeds—can improve generative performance and training efficiency.

The authors formulate asynchronous flow matching for composite latents, specifically separating semantic and texture representations. They define a global time variable to coordinate the sampling trajectory while allowing each representation to follow its own local time. To optimize this, they introduce a schedule-corrected objective that ensures the weighting of local noising times remains consistent as the schedule is updated. They also propose a parametric schedule class that is convex and monotone by construction, enabling efficient joint optimization of the schedule and the denoiser with minimal additional compute.

The learned asynchronous schedule provides significant empirical improvements over hand-tuned baselines. On ImageNet 256x256, the authors' 675M-parameter model achieves an FID of 1.05 in just 200 epochs, matching a much larger baseline that required 800 epochs. With further training, the model reaches an FID of 1.02, outperforming a 1B-parameter model while using fewer parameters. The method also demonstrates robustness by improving performance across different semantic representation choices, such as DINO-PCA and CLIP-PCA, confirming that the benefits stem from the learned schedule rather than specific architectural tricks.