ResearchPod Summary
Knowledge distillation (KD) is essential for compressing large diffusion models, but it often struggles to transfer knowledge across the entire conditioning space, especially when paired image-text data is limited. While previous methods like Random Conditioning (RC) address this by exposing students to a broader range of text prompts, they treat all conditions as equally informative. This paper asks: once a large pool of auxiliary conditions is available, how should training effort be optimally allocated to maximize student performance?
The authors introduce ARIA (Adaptive Region-based Importance Allocation), a framework that partitions the conditioning space into coarse regions. Instead of scoring individual samples—which is computationally expensive—ARIA maintains online estimates of teacher-student discrepancy at the region level using an exponential moving average (EMA). During training, the model adaptively samples conditions from regions with higher tracked discrepancy, ensuring that the student receives more supervision in areas where it currently performs poorly. This approach is modular and can be integrated into existing distillation pipelines without modifying the underlying learning objective.
Empirical evaluations across multiple architectures (Stable Diffusion v1.4, v2.1, and SDXL) demonstrate that ARIA consistently outperforms uniform random conditioning. The gains are particularly significant in unseen or underrepresented regimes, where the student model might otherwise fail to generalize. The authors also provide a theoretical analysis establishing that the EMA-based tracking mechanism effectively follows evolving discrepancy under bounded variance and drift, providing a formal guarantee for the stability of the adaptive sampling process.
ARIA provides a scalable, efficient way to improve model compression in data-constrained settings. By focusing training on "difficult" regions of the conditioning space, it enables smaller student models to achieve higher fidelity and better alignment with the teacher model. This is particularly valuable for practitioners working with large-scale text-to-image systems where generating synthetic training data is computationally prohibitive.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.