Eigenvalue Calibration for Semantic Embeddings of Large Language Models

As large language models (LLMs) are increasingly used in high-stakes decision-making, quantifying their uncertainty has become critical. Recent methods use the eigenvalues of semantic embeddings to measure confidence. However, traditional calibration techniques designed for classification probabilities do not directly apply to these eigenvalue-based measures. This paper addresses this gap by developing a formal framework for calibrating density matrix predictors and their eigenvalues.

The authors propose interpreting LLMs as density matrix predictors, where the model output is represented by the expected outer product of the semantic embeddings of generated answers. They define a novel notion of matrix and eigenvalue calibration and establish an entropy-risk equivalence, showing that under proper calibration, the expected uncertainty (entropy) matches the model's risk. To improve reliability, they introduce a temperature scaling method for eigenvalues—analogous to standard probability calibration—and prove that this approach optimizes calibration when minimizing proper score risks.

Experiments across several LLMs (Phi-4, Phi-4 Mini, Llama4 Maverick) and datasets (TriviaQA, Natural Questions) demonstrate that current models are systematically overconfident. The authors show that applying their temperature scaling method effectively mitigates this overconfidence, leading to better-calibrated uncertainty estimates. Their theoretical results confirm that minimizing the matrix version of the cross-entropy risk is a robust way to determine the optimal temperature parameter, which consistently improves the reliability of the model's uncertainty quantification.

This work provides a rigorous mathematical foundation for uncertainty quantification in LLMs using semantic embeddings. By enabling more reliable confidence estimates, this framework helps developers and researchers better identify when an LLM is likely to be incorrect, which is essential for the safe and reliable deployment of AI systems in real-world applications.