Uncertainty-based Debiasing and Unlearning for Decontamination

Data contamination—where benchmark test samples appear in a model's training corpus—inflates performance metrics and prevents fair evaluation of LLM capabilities. Current decontamination methods often rely on aggregate accuracy, which masks per-sample behavior, or require access to an uncontaminated reference model. This paper addresses the need for a sample-level decontamination framework that does not require external clean models or prior knowledge of which specific samples are contaminated.

The authors propose a sample-level evaluation framework using distributional distance metrics (KL-divergence and L1-difference) to compare a decontaminated model's output distribution against an uncontaminated reference. To perform decontamination, they introduce Uncertainty-Based Decontamination (UBD). UBD leverages a deep ensemble of contaminated models to estimate epistemic uncertainty. Because memorized samples are sensitive to training batch ordering, they exhibit higher variance across ensemble members. The authors use this uncertainty to calculate a per-sample correction scalar, which is then applied either as a post-hoc output debiasing step or as a soft training signal for model unlearning.

Experiments on MMLU-Pro and MATH-MCQA demonstrate that UBD significantly outperforms existing baselines like paraphrasing or choice-permutation. UBD produces output distributions that are substantially closer to those of an uncontaminated model, effectively reducing the inflated probability mass on correct answers caused by contamination. Crucially, the method achieves this while maintaining model performance on uncontaminated data, proving it is a robust, practical tool for real-world benchmarking.

This work shifts the focus of decontamination from simple accuracy metrics to distributional fidelity. By providing a way to "clean" a model's outputs without needing a separate, uncontaminated version of the model, UBD offers a scalable solution for researchers and developers to ensure that reported benchmark results reflect genuine model intelligence rather than rote memorization.