Anh Nguyen, Ngan Nguyen, Duc Vu, Trung Dao, Viet Nguyen, Quan Dao, Kien Nguyen, Chi Tran, Phong Nguyen, Khoi Nguyen, Cuong Pham, Dimitris Metaxas, Vishal M. Patel, Anh Tran
5 min
Abstract
Modern one-step diffusion models achieve impressive quality through distribution-based timestep distillation. Yet, they rely on a critical assumption: Teacher and Student must inhabit the same latent space. This Shared-Space constraint prevents knowledge transfer from modern high-capacity Teachers (e.g., SD 3.5 and Flux) into compact, deployment-friendly Students such as SD 1.5, whose latent resolution and VAE parameterization differ from the Teacher. We formalize this overlooked regime as Cross-Space Distillation, where Teacher and Student differ in both latent resolution and VAE space. To enable distillation under this mismatch, we introduce the Bridge, a lightweight latent interface that maps Student latents into the Teacher space without modifying the Student backbone. Bridge combines a frozen Student VAE decoder as a spatial prior with a compact learnable projector, and is trained with latent reconstruction and attention fidelity objectives for stable Teacher-space alignment. Across diverse modern Teachers, Bridge enables substantial gains for compact one-step Students; for example, it improves SD 1.5 from 5.4 to 9.4 HPSv3 while preserving one-step inference, low latency, and broad ecosystem compatibility. These results show that heterogeneous large Teachers can be distilled into efficient, deployable backbones through a lightweight latent-space interface.
Sam: The team built a small, learnable module—think of it as a universal translator sitting between the student and the teacher. The student model does its normal thing and produces its own compact shorthand. Before that shorthand ever reaches the teacher, the Bridge intercepts it and converts it into the format the teacher understands. So the teacher can now give useful feedback, and the student can actually learn from it.
Alex: So the student never has to change. It just gets a translator bolted on the side.
Sam: Exactly. And the Bridge itself is lightweight—it doesn't add much computational cost. There's one part that preserves the spatial structure of the image, making sure that when the translation happens, the positions and shapes within the image stay accurate. And there's a small trainable component that handles the actual format conversion.
Alex: How do they make sure the student is actually learning the right things from the teacher, and not just superficially mimicking it?
Sam: That's where a technique they call "Attention Fidelity Loss" comes in. Here's the idea: when an AI model looks at an image, it doesn't pay equal attention to every part of it. It focuses more on certain regions—the face in a portrait, the edges of an object, the area where two textures meet. You can visualize this as a kind of heatmap laid over the image, showing which parts the model is prioritizing. Researchers call these "attention maps." The Attention Fidelity Loss compares the student's heatmap to the teacher's heatmap and penalizes the student whenever they differ significantly. Over time, this pushes the student to notice the same things the teacher notices—to develop the same sense of what matters in an image.
Alex: So it's not just about copying the teacher's output. It's about learning to see the way the teacher sees.
Sam: That's a good way to put it. And the practical result is that a compact model like Stable Diffusion 1.5—which runs on relatively modest hardware—can meaningfully improve its image quality by learning from a much more capable modern teacher. That's something that wasn't straightforward before this approach.
Alex: So the insight isn't "build a bigger model." It's "build a better translator so the smaller model can learn from the best available teacher."
Sam: That's the core of it. And it's a useful reminder that progress doesn't always require redesigning the system from the ground up. Sometimes the more practical path is improving how knowledge moves between systems that already exist. Thanks for listening to ResearchPod.