Ensemble Diversity Optimization for Subjective Supervision

Subjective NLP tasks—such as content moderation or sentiment analysis—often involve genuine disagreement among annotators. Standard approaches typically aggregate these labels into a single ground truth, which discards valuable distributional information and leads to overconfident, poorly calibrated models. The authors propose Ensemble Diversity Optimization (EDO), a framework designed to preserve this epistemic uncertainty by treating annotator disagreement as a meaningful signal.

EDO operates in the prediction space using an ensemble of pre-trained models. It introduces a unified differentiable objective that balances three competing goals: predictive utility (via a soft F1 surrogate), calibration (via class-weighted cross-entropy), and internal ensemble diversity. A key innovation is the signed diversity regularizer, which allows researchers to either preserve disagreement (to capture subjectivity) or suppress it (to mitigate structural noise or imbalance). The framework also uses Gumbel-Softmax relaxation to learn the optimal ensemble size end-to-end.

Experiments across four subjective text-classification benchmarks (ArMIS, ConvAbuse, HS-Brexit, and MD-Agreement) demonstrate that EDO consistently outperforms traditional baselines. By explicitly optimizing for diversity and calibration, EDO achieved substantial improvements in probabilistic calibration, reducing cross-entropy by 40–78% depending on the baseline. Furthermore, the model maintained competitive F1 scores while showing better alignment with the underlying annotator distributions compared to methods like Soft-CE, Soft-MD, and Top-5 Voting.

This work provides a principled, model-agnostic way to handle subjective supervision without requiring complex annotator metadata. By moving away from the assumption of a single latent ground truth, EDO allows models to represent the full spectrum of human interpretation. The ability to control the diversity-calibration trade-off via a simple signed hyperparameter makes this approach highly flexible for practitioners dealing with ambiguous or imbalanced datasets.