Tong Zhao, Xianfang Zeng, Yucheng Han, Ruoyu Wang, Mingkun Lei, Gang Yu, Chi Zhang
8 min
Abstract
Distribution matching distillation (DMD) has emerged as a leading paradigm for distilling diffusion models into few-step students, where an auxiliary fake score model tracks the student's distribution and the discrepancy between the teacher and fake predictions drives the student toward the teacher. In multi-step distillation, although only the student's final output is delivered, existing methods train the fake model on intermediate states or across all denoising steps. Consequently, the student's update signal becomes entangled with intermediate distributions, which can distort the matching direction and remain nonzero even after the student's final outputs already match the teacher distribution. To resolve this mismatch, we propose Endpoint Distribution Matching Distillation (eDMD), which trains a single shared fake model exclusively on the student's final outputs and uses it to correct every denoising step. Our theoretical analysis proves that this endpoint reference guarantees that the distribution matching term vanishes if and only if the student's final output distribution matches the teacher's, regardless of intermediate distributions. Beyond the fake reference, we further strengthen teacher supervision through a precise teacher (PT) that replaces single-step guided predictions with short multi-step trajectories on the final noise interval, paired with a calibrated query density for intermediate steps. Experiments on SD3-Medium, FLUX.2-klein (4B and 9B), and Qwen-Image-2.1 demonstrate that eDMD consistently outperforms existing distribution matching methods for 4-step generation.
Alex: So the destination can be right even when the route looks different. How do the authors make that more than an intuition?
Sam: They split the correction signal into three parts. One comes from enhancing the teacher’s prediction. Another measures the mismatch between the teacher and the student’s finished outputs. The last is an unwanted residual caused by training the fake model on a different reference distribution.
Alex: If the student’s finished outputs already match the teacher, that middle part should disappear. Does the unwanted part survive?
Sam: It can. Under exact denoisers—models returning the true average clean sample given a noisy observation—the authors prove that the residual vanishes precisely when the fake model represents the endpoint distribution. That means the distribution of finished outputs, regardless of what happens between steps.
Alex: What does that failure look like in something concrete enough to inspect?
Sam: They use a synthetic, tree-shaped distribution in a plane. They construct students whose final samples match the teacher, but whose intermediate predictions or states differ. The intermediate-based references still move those final samples away from the teacher. The endpoint reference gives no correction.
Alex: That isolates the reference choice from image aesthetics. But does using the endpoint mean training only the final generation step?
Sam: No, and that distinction matters. The fake model learns exclusively from final outputs, but it supplies corrections to every student step. Training selects a step and sends gradients through that step’s prediction, not through the entire remaining generation chain. Restricting student updates to the final step performed much worse in their tests.
Alex: Does this mean every local correction is the true gradient of final image quality?
Sam: No. The appendix explicitly distinguishes those local updates from gradients propagated through the later steps. Endpoint referencing removes the unwanted reference residual. It does not make every intermediate update identical to directly optimizing the final output through the whole chain.
Alex: There are also teacher changes in the method. What do they add beyond fixing the student reference?
Sam: A single denoising prediction can average over plausible outcomes. In the tree example, it can land between branches. Their precise teacher instead follows a short denoising trajectory for low-noise queries at the final student step. That gives targets that better preserve the tree’s fine structure.
Alex: Does that extra teacher work also make the deployed student slower?
Sam: It adds teacher evaluations during training, not student inference. They also calibrate where intermediate steps query the teacher. Before distillation, they measure consistency across noise levels and favor levels with more reliable supervision. This helped more on the teacher with stronger prediction drift.
Alex: Let’s turn to the evidence. How broad was the image-generation test?
Sam: They distilled four teacher backbones, covering SD3-Medium, both tested FLUX variants, and Qwen-Image. Training used prompts and generated samples, without real images. Their main DrawBench evaluation generated a thousand images from two hundred prompts.
Alex: What is the clearest comparison, and what does the score actually tell us?
Sam: On FLUX’s smaller tested backbone, eDMD reached about 12.7 on HPS version three plus-plus. DMD2, the strongest distribution-matching baseline there on that metric, reached about 11.3. This is an automated human-preference score, not a direct count of people choosing images in this study.
Alex: Since the full method changes both the reference and the teacher targets, how much can we attribute to the endpoint idea?
Sam: Removing the endpoint reference produced the largest overall degradation among their component ablations. On that same backbone, the score fell to about 11.1. They also inserted endpoint referencing into their TDM implementation while retaining its student objective and rollout, and performance improved across the tested teachers.
Alex: Peak-checkpoint comparisons can flatter a method if selection and evaluation use the same prompts. Did they examine that?
Sam: They did. Splitting prompts into selection and held-out halves preserved eDMD’s top ranking across all tested backbones. That strengthens the comparison, though it remains an internal check within the same benchmark. Some baselines were adapted or reimplemented, and their DMD2 comparison omitted its adversarial loss.
Alex: And the theorem does not automatically guarantee that the enhanced teacher targets reproduce the original teacher distribution?
Sam: Correct. Residual cancellation holds for any teacher target when the fake model is exact. But the stronger matching guarantee requires targets consistent with one clean distribution. Guided predictions and trajectory-based targets generally do not meet that condition. The authors explicitly distinguish these claims.
Alex: So who should read the full document, and where should they begin?
Sam: Researchers building diffusion-distillation methods should start with “Reference distribution of the fake model.” Then read the component ablations and the endpoint-reference transfer to TDM. Together, those separate the mathematical mechanism from the benefits of the full training recipe.
Alex: And for researchers deciding whether the result travels beyond image generation?
Sam: Treat it as strong evidence for this reference choice in the tested image setting, not a demonstrated result for video or audio. Also, endpoint-based choices existed in earlier implementations; this work explains the mismatch and combines endpoint training with corrections at every step. The line to remember is: matching finished outputs need not mean matching every unfinished prediction.
Alex: Thanks for listening.