A Mechanism-Driven Theory of Phase Transitions in Active Learning

Active learning (AL) is a strategy to reduce annotation costs by iteratively selecting the most informative samples for labeling. While it is well-known that different strategies—such as those based on representativeness, coverage, or uncertainty—excel at different stages of the labeling process, these stages have historically been defined by arbitrary, heuristic label counts. This paper reframes AL budget regimes as structural phase transitions, where the dominant mechanism governing generalization shifts as the labeled set grows.

By reinterpreting PAC-style generalization bounds, the authors decompose the true risk into four interacting components: empirical risk, distributional discrepancy, model complexity, and a statistical confidence term. The paper proves that as a model is trained on an adaptively sampled dataset, the relative influence of these components is structurally unstable. This creates a moving bottleneck for generalization, explaining why no single acquisition strategy remains optimal throughout the entire learning trajectory.

To operationalize this, the authors introduce measurable proxies for each bound component and use a segmented regression framework to identify three distinct phases:

1. Data-Driven Phase: Dominated by distributional discrepancy, where representativeness-based strategies (e.g., TypiClust) excel by aligning the labeled subset with the underlying data distribution.
2. Transition Phase: Governed by geometric coverage and hypothesis-space complexity, where coverage-oriented methods (e.g., Coreset) effectively expand the model's support across the feature manifold.
3. Model-Driven Phase: Ruled by empirical risk refinement, where uncertainty-based methods (e.g., Entropy) dominate by focusing on informative boundary samples to reduce residual training loss.

This framework provides a principled, transition-aware basis for designing the next generation of AL algorithms. By monitoring proxy dynamics, future systems can automatically switch acquisition strategies as the dominant bottleneck evolves, rather than relying on static, heuristic-based schedules. Furthermore, the authors demonstrate that self-supervised representation learning compresses these phases, shifting transition points earlier in the labeling trajectory and highlighting the critical synergy between representation quality and adaptive sampling efficiency.