Data Selection Through Iterative Self-Filtering for Vision-Language Settings

As vision-language models scale, training on noisy, internet-scale datasets becomes a major bottleneck. While data pruning is common, it typically relies on expensive heuristics, curated reference datasets, or pre-trained models. The authors investigate whether a model can effectively filter its own training data during the learning process, thereby improving sample efficiency and downstream performance without external dependencies.

The proposed Self-Filtering method treats data selection and model training as two facets of the same optimization problem. The process iterates through three phases:

1. Training: The model is trained on a mixture of the full dataset and a subset of 'likely clean' samples.
2. Filtering: The model evaluates the entire dataset, scoring image-text pairs based on the cosine similarity of their embeddings.
3. Updating: The dataset is refreshed by oversampling the highest-scoring pairs, creating a new, improved mixture for the next round.

Crucially, the method avoids overfitting to 'easy' samples by maintaining a mix of the full distribution. This allows the model to gradually learn harder, more complex samples as its internal representation improves over successive rounds.

Experiments on the DataComp small-scale benchmark demonstrate that Self-Filtering consistently outperforms models trained on the full, unfiltered dataset. By iteratively refining the data mixture, the model achieves higher zero-shot accuracy and retrieval performance across 38 downstream tasks. The authors show that this self-improving loop is more sample-efficient than static filtering approaches, as it adapts the data selection to the model's evolving understanding of the visual-language alignment.

This work provides a path toward training performant multimodal models in resource-constrained environments. By removing the reliance on pre-trained models for data curation, researchers can train models that are less biased by the specific limitations of existing, proprietary filtering pipelines. It also highlights the potential for models to act as their own curators, turning the challenge of noisy data into an opportunity for self-supervised improvement.