Yuxiang Fu, Qi Yan, Zike Wu, Yongxing Zhang, Purang Abolmaesumi, Lele Wang, Renjie Liao
9 min
Abstract
Flow and diffusion models suffer from slow inference due to computationally expensive numerical integration. Distillation provides a promising way for a student model to learn from a teacher's dynamics, enabling one-step or few-step generation. However, existing methods often depend on curated distillation datasets, costly teacher rollouts, or auxiliary proxy networks, which complicate model training and scaling. In this work, we propose Consistent Distribution Matching, a simulation-free and data-free distillation method for accelerating diffusion and flow models while preserving strong generative capacity. Our key insight is to unify sample generation and score estimation with one student network. Thus, our framework uses only two models, a frozen teacher and a trainable student, and optimizes one objective. We prove that minimizing our objective indicates Wasserstein convergence of the student flow-map pushforwards to the teacher marginals. On ImageNet 256$\times$256, our method attains an FID of 2.04 with a single function evaluation (1-NFE) and a 4-NFE FID of 1.37 within 40 epochs of training, surpassing the state-of-the-art distillation baselines without data. Our code code and model are available at https://consistentdmd.github.io/.
Alex: Here’s the central move. The student predicts average velocity: total movement divided by the duration of an interval. That lets it jump between different stages of generation, rather than only making a complete noise-to-image jump.
Sam: A large jump gives you an image. But the teacher normally tells you how to move locally. How do those predictions meet?
Alex: The authors derive a local velocity estimate from the student’s average-velocity prediction and its rate of change. The same student therefore generates samples and supplies the estimate previously assigned to a helper network. Training makes that corrected estimate agree with the teacher’s local velocity.
Sam: Walk through a training example. Where does the state being corrected actually come from?
Alex: Start with random noise. The student generates a clean image, then the training procedure adds noise back using a known formula. That produces an intermediate state where the teacher can say how to move.
Sam: So the teacher corrects states relevant to what the student currently makes, rather than states borrowed from a separate dataset.
Alex: Yes, although those states need not initially resemble the teacher’s own states. Another construction lets the student jump directly from noise to an intermediate stage. Mixing these constructions performed better than relying on either alone in their ImageNet ablation.
Sam: There is still a derivative inside that correction. Does eliminating a helper network simply move the difficulty somewhere else?
Alex: It leaves a real technical issue. They compute a Jacobian-vector product, which measures how the prediction changes along a chosen direction. They detach this correction during training, so optimization does not differentiate through it again.
Sam: And the direction matters because the student and teacher might disagree about where to go.
Alex: They test that on synthetic spiral and checkerboard distributions. Using the teacher’s local velocity gives the smallest, most stable correction statistics. Using the student’s full noise-to-image displacement produces substantially more variability.
Sam: Before the image scores, what does the mathematical result establish? Self-generated training states could miss whole regions.
Alex: The theorem needs coverage assumptions addressing that concern, alongside smoothness and integrability conditions. Under those assumptions, population training loss approaching zero implies the student’s generated distributions approach the teacher’s, averaged over reference times. The distance used measures how far probability mass must move.
Sam: That is conditional convergence, then, not a promise that every practical training run finds the teacher.
Alex: Correct. A second result identifies the teacher’s average velocity at a global optimum, provided the student can represent it and the coverage conditions hold. The synthetic spiral experiment supports the distinction: CDMD and FreeFlow match the final distribution similarly, but CDMD learns the teacher’s transport more accurately.
Sam: Let’s turn to the practical test. What is the strongest image-generation comparison?
Alex: On class-conditional ImageNet, they compare methods under matched training budgets and a shared evaluation pipeline. Their Fréchet Inception Distance, or FID, compares generated and reference image distributions; lower is better. They evaluate it using fifty thousand generated images.
Sam: With the same teacher, how much separates the proposed student from an established baseline?
Alex: With LightningDiT as teacher, CDMD gets a one-call FID of 2.04, compared with 3.43 for DMD2. It also achieves the best one-call FID among the compared data-free methods with each of the other teachers.
Sam: And if you can afford extra calls, do you need another round of training?
Alex: Not for that student. With four calls, its FID reaches 1.37, compared with the teacher’s 1.35. That is close on this metric, not proof that their images are interchangeable in every respect.
Sam: Does that improvement carry over automatically to text-to-image generation?
Alex: They test one large text-to-image teacher, SANA. CDMD leads the learned few-step methods on aesthetics, PickScore, ImageReward, and FID. But it trails several methods on human-preference and text-alignment metrics, so the evidence is mixed rather than a clean sweep.
Sam: The headline says simulation-free. Does that describe the larger implementation without qualification?
Alex: The text-to-image implementation needs a qualification. It caches intermediate states along a coarse student generation trajectory, then adds noise from a nearby cached state. It still avoids teacher rollouts, but it is not simply the single-jump training-state construction.
Sam: Does that more involved version retain a training-cost advantage?
Alex: On their matched hardware, CDMD takes about seven GPU hours, compared with thirty for DMD2. Consistency distillation is cheaper still, though it scores lower on the reported preference measures.
Sam: What limitation would you put beside those results before someone adopts the method?
Alex: The derivative correction remains insufficiently constrained, and the authors report large, fluctuating norms during training. Extra sampling steps can also worsen quality, depending on guidance, the sampling rule, and exposure to local intervals. The text-to-image results do not establish statistical superiority.
Sam: Given those conditions, who should read the full document, and where should they start?
Alex: Researchers developing diffusion distillation should start with “Minimal Design on Data-Free Distillation,” then the matched ImageNet comparison. Read the theorem’s coverage assumptions and the text-to-image implementation details before generalizing the headline. For everyone else, remember this: one student can learn both to generate images and to supply the velocity estimate used to correct itself.
Sam: A simpler pipeline, with conditions worth checking.