The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Models

Research Context and Objective

Large Language Models (LLMs) are increasingly used for complex reasoning and content generation, yet their inherent stochasticity—the unpredictable variability in their outputs—poses a significant barrier to reliability. While researchers have long used the concepts of aleatoric (data-driven) and epistemic (model-driven) uncertainty to understand these systems, these categories are often too broad to diagnose specific failures in modern transformer-based architectures. This paper aims to bridge the gap between theoretical uncertainty origins and practical deployment by proposing a granular taxonomy and a comprehensive evaluation framework for Uncertainty Quantification (UQ) methods.

A Granular Taxonomy of Uncertainty

The authors propose a new framework that decomposes LLM uncertainty into four distinct sources: input-level (prompt sensitivity), parameter-level (model weights), token-level (distributional variance), and decoding-process (generation strategy). By mapping these sources, the authors categorize 21 common UQ methods into four groups: Bayesian, ensemble, consensus-based, and single-pass approaches. This classification allows developers to select UQ techniques based on model accessibility (white-box vs. black-box), training requirements, and computational cost.

Empirical Evaluation and Findings

The researchers conducted a large-scale empirical study using three prominent LLM families (Qwen3, Llama 3.2, and DeepSeek-V3) across diverse benchmarks including TriviaQA, GSM8K, and HumanEval. Their findings reveal three critical insights:

1. The performance of UQ methods is highly dependent on the specific task and generation setting.
2. Consensus-based methods, particularly those labeled Deg (Degree) and EigV (Eigenvector), provide the most robust performance across different models and objectives.
3. There is a clear empirical scaling law: as model scale increases, the estimated uncertainty generally decreases, suggesting that larger models are inherently more stable in their predictive outputs.