The task of dataset distillation concerns the learning of a small set of synthetic samples that can serve as an effective replacement for a large real dataset when used for downstream tasks. While many solutions have been proposed since the problem's inception, most of them primarily seek to optimize the distilled data with respect to a classifier. In this work, we propose a new distillation method that learns a set of samples such that they directly approximate the per-class data distribution. Specifically, we "drift" our distilled data in the latent space of a feature extractor such that they move both toward the real data and away from each other without using a classification objective. We find that our drifted data cover the real distribution significantly better than existing methods while still serving as powerful training samples for classification. Furthermore, we showcase our method's use as a visualization tool for better interpreting models and extend it beyond images to other modalities. Our work advances the frontier of dataset distillation, offering a novel synthesis method and proposing distribution-based metrics as a complementary evaluation. More broadly, our findings help address the growing need to understand both data and models by uncovering more about how they interact.
Sam: Tiny synthetic datasets can train classifiers better without being optimized to please a classifier. An image-distillation study finds that representing the variety within each class can beat directly chasing classification performance.
Alex: That sounds like a change in what counts as a good compressed dataset. Is the claim better training, or a more faithful picture of the original data?
Sam: Both, in the settings tested. This is a method paper called “Dataset Distillation via Drifting,” by George Cazenavette and colleagues at MIT. They learn a small synthetic replacement for a large dataset, then test its usefulness and how broadly it represents the source.
Alex: So this matters if I build compact training sets, but also if I use synthetic examples to understand a model. Let’s unpack what changes, what the comparisons establish, and where the claim stops.
Sam: Start with the usual goal. Dataset distillation means learning a tiny set of artificial examples that can replace a much larger training collection. Unlike selecting representative real examples, you can change the synthetic examples themselves.
Alex: If those examples are meant to train a classifier, optimizing them for classification seems sensible. What do the authors think that leaves out?
Sam: A classifier needs to separate classes, not necessarily represent all the variation within them. The authors argue that optimizing around its decision boundary can bake its biases into the synthetic data. That may limit usefulness for other tasks or models.
Alex: So an image could be useful for teaching “butterfly, not bird” without representing much of what butterflies look like. What replaces that objective?
Sam: They borrow an approach called drifting from generative modeling. Each synthetic example is pulled toward real examples and pushed away from other synthetic examples. The attraction encourages resemblance; the repulsion encourages diversity.
Alex: But “toward” needs a distance measure. Comparing raw pixels would treat a shifted object very differently, even if its meaning stayed the same.
Sam: They measure distances in a feature space: a pretrained model’s numerical representation of an image. They update the synthetic images so those representations move under the competing forces. They optimize each class separately, without a classification loss.
Alex: That still leaves a severe compression problem. How can a handful of pictures represent a whole class?
Sam: The crucial move is that the stored pictures do not have to do it alone. The method also optimizes augmentations, meaning transformed views such as random crops and horizontal flips. Multiple views of each synthetic image participate in the drifting process together.
Alex: So different crops of the same image can represent different parts of the class distribution. Is that encouraged explicitly, or does it just happen?
Sam: It is encouraged explicitly. Views from the same synthetic image also repel one another. In the authors’ visualization, one stored image has augmented views spread across several regions occupied by real data. The target is the distribution of images together with their transformations.
Alex: That makes the transformations sound central, rather than routine training decoration. Did they test whether the method works without them?
Sam: They did. With a deterministic resized view instead of random augmentation, classification performance and coverage collapsed. Adding more jointly optimized views improved every reported metric across the tested settings. So the evidence supports this whole construction, not attraction and repulsion alone.
Alex: Before the results, is there another ingredient someone reproducing this would need?
Sam: Yes: how the images are represented during optimization. They use a Laplacian pyramid, meaning layers at different resolutions, which are enlarged and added together. They start coarse and progressively add detail, and also apply a color transform that favors natural colors.
Alex: Then this is not simply an interchangeable loss applied to raw pixels. Does that representation materially affect the outcome?
Sam: Replacing the pyramid with raw pixels sharply reduced training performance and coverage. The pyramid acts as implicit regularization, discouraging the adversarial-looking patterns that can appear when optimizing inputs against a model.
Alex: Let’s put the headline result on solid ground. What was trained, and against which comparison?
Sam: The main benchmark compresses ImageNet, a thousand-category image dataset, and trains a fresh residual-network classifier. With ten synthetic images per class, the strongest noise-initialized drifting variant reached about twenty-seven percent accuracy using ordinary class labels. The best noise-initialized baseline reached about twelve percent.
Alex: That is a large gap, but evaluation choices can change these rankings. Did the authors retrain the competing methods under the same recipe?
Sam: They re-evaluated every method with identical training recipes and reported results across three fixed random seeds. They also separated synthetic data initialized from noise from data initialized using real images. Drifting performed strongly under both initializations.
Alex: The paper also distinguishes fixed labels from knowledge distillation. Why does that matter for interpreting a tiny dataset’s usefulness?
Sam: With knowledge distillation, a teacher model supplies a fresh target for each newly augmented training image. The authors argue that this measures teacher-to-student transfer rather than the original from-scratch task. They therefore also test ordinary class labels and soft targets computed once before training.
Alex: So a student can start with fresh weights in every case, but a continuously consulted teacher supplies extra information. The fixed-target tests ask a different question.
Sam: And the strongest training data came from feature extractors trained without labels, rather than their supervised classifier. That suggests rich representations matter more here than matching the student’s architecture. It does not establish the same ranking for every possible student.
Alex: Training accuracy is only half the argument. How do they measure whether these synthetic images represent more of the original data?
Sam: Their main distribution measure is coverage. Around each real validation example, they define a neighborhood using distances to other real examples. They then ask whether that neighborhood contains a synthetic example or one of its crops, in a pretrained model’s feature space.
Alex: So coverage is not a literal percentage of all possible natural images. It is proximity to held-out examples, measured through a particular model.
Sam: Yes. In one independent evaluation space, their noise-initialized DINO variant covered seventy-seven percent of those neighborhoods. The strongest noise-initialized baseline covered roughly ten percent. That gap was much larger than the gaps on classification tests performed within feature space.
Alex: If the method optimizes distances inside a model, evaluating with that same model could flatter it. Do they address that?
Sam: They flag those self-evaluations and exclude them from rankings. Coverage was substantially inflated when the distillation and evaluation models matched. My reading is that cross-model results strengthen the evidence, but coverage remains a model-dependent measure, not proof of matching the full natural-image distribution.
Alex: And does the method beat every competing way of making synthetic data?
Sam: No. The appendix compares methods using pretrained diffusion generators. Drifting is competitive on fixed soft-label training, but falls behind leading diffusion methods in some other training settings. Its advantage is that it does not require a pretrained generator.
Alex: There is also the question of whether a tiny resulting dataset means a cheap process overall.
Sam: Not necessarily. The authors estimate roughly two hundred and fifty graphics-processor hours for the main dataset at the smaller compression setting, before broader experiments. Classes can be processed in parallel, but they explicitly call the total compute significant.
Alex: Does the paper give reasons to read beyond the image benchmark?
Sam: It extends the approach to music and action videos, beating a representative-subset selection baseline on both. It also shows that different feature extractors produce visibly different synthetic images. Those are useful demonstrations, though the audio and video comparisons are narrower than the image evaluation.
Alex: Who should make time for the full document, and where should they start?
Sam: Researchers working on dataset distillation should start with “Distribution Matching,” then read the evaluation protocols and the appendix comparison with diffusion-based methods. Those explain both the conceptual change and its competitive limits. For representation researchers, “Distilled Data Elucidate Model Biases” is the most relevant extension.
Alex: And what should someone outside this area carry back to their colleagues?
Sam: A tiny dataset can learn more by representing variation than by chasing a classifier’s decisions. This study makes that case with strong benchmark comparisons, while showing that the representation used to judge variety still matters.
Alex: Thanks for listening.