Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data

Researchers frequently use machine learning and large language models (LLMs) to classify text, reporting metrics like precision, recall, and F1 to validate their models. However, these metrics are point estimates subject to sampling variation. Despite this, researchers often fail to report measures of uncertainty, or they use statistical methods that are inappropriate for the small sample sizes, high performance levels, or nested data structures (e.g., multiple texts from the same author) common in social science research.

This paper evaluates various methods for calculating confidence intervals for classifier performance metrics. The author uses simulations to compare the coverage accuracy of established analytic methods (Wald, Wilson, Agresti-Coull, Clopper-Pearson) and bootstrap methods (percentile, BCa, and a proposed pseudo-count regularized bootstrap). The study specifically examines performance under conditions of small-to-moderate sample sizes, infrequent constructs, and nested data structures.

• Analytic Methods: The commonly used Wald interval performs poorly, often providing coverage significantly below the 95% target. The Agresti-Coull, Wilson, and Clopper-Pearson methods are much more reliable.
• Bootstrap Methods: The basic percentile bootstrap is often inaccurate for small samples. The author introduces a pseudo-count regularized bootstrap—which adds artificial successes and failures to the sample—to improve the stability and accuracy of intervals for metrics like F1.
• Nested Data: When texts are nested within individuals, standard intervals fail. Accurate estimation requires adjusting for effective sample size and degrees of freedom. The hierarchical bootstrap is generally superior to the cluster bootstrap, provided the number of texts per individual is not extremely small.