Position: Unlabeled̸= No Human Supervision in Visual Learning

In modern computer vision, the term "unsupervised" has become a catch-all for any method that does not use explicit human-provided labels. However, this position paper argues that this label is increasingly misleading. While these models do not use traditional annotations, they are deeply shaped by human choices—ranging from how datasets are curated and filtered to the specific invariances encoded in training objectives. As the field shifts toward foundation models, these human priors are becoming more pervasive yet less visible, creating a conceptual ambiguity that hinders scientific progress.

An analysis of flagship computer vision conferences (CVPR, ICCV, ECCV) reveals a striking trend: the explicit use of the terms "unsupervised" and "self-supervised" in paper titles peaked around 2021 and has since declined, despite the continued growth of the field. Full-text analysis confirms this is not just a naming convention shift; it reflects a deeper change in how research is conducted. Increasingly, papers labeled as "unsupervised" are actually building on top of a small, concentrated set of pre-trained backbones (e.g., DINO, CLIP, MoCo). This reliance on pre-trained infrastructure means that many "unsupervised" methods are effectively inheriting the biases and inductive priors of those upstream models.

This lack of clarity poses several risks to the research community:

• Misattribution: Researchers may credit a new mechanism for emergent properties that are actually inherited from pre-trained representations.
• Distorted Incentives: The field may prioritize incremental adaptations of existing foundation models over the exploration of fundamentally new learning principles.
• Reduced Diversity: Methods that do not rely on massive, pre-trained scale may struggle to compete on benchmarks, potentially marginalizing early-stage research into alternative supervision signals like motion or physical interaction.