Conformalized Large Language Models under Configuration Shift

As Large Language Models (LLMs) are increasingly used in high-stakes environments, researchers use Conformal Prediction (CP) to provide statistical guarantees that a model's output set contains the true answer. However, CP typically assumes that the scoring mechanism is fixed. This paper investigates whether changing inference-time configurations—such as prompt templates, decoding temperatures, or weight quantization—breaks these guarantees, a phenomenon the authors term "configuration shift."

The authors conduct a systematic empirical study across 9 LLMs, 4 datasets, and 4 nonconformity scores. They measure two primary outcomes: validity (the probability that the true answer is in the prediction set) and efficiency (the size of the prediction set). They compare these metrics between an in-distribution baseline (where calibration and test configurations match) and a shifted setting (where they differ). Additionally, the authors derive theoretical lower bounds on coverage based on the discrepancy between calibration and test score distributions, providing diagnostic tools to quantify shift severity.

Configuration shift consistently causes undercoverage, meaning the model fails to meet its promised confidence level. While the size of the prediction sets (efficiency) remains relatively stable, the validity of the models degrades significantly. The authors find that decoding temperature is the most damaging factor, followed by prompt templates and quantization. Crucially, increasing the amount of calibration data does not fix this issue; instead, it exposes the bias caused by the discrepancy between the calibration and test score distributions. The authors propose practical mitigations, including bound-inspired recalibration and fragility-aware calibration ensembling, to recover lost coverage.

This work highlights that conformal guarantees in LLMs are fragile. Because practitioners frequently adjust prompts or model settings after calibration, they may inadvertently invalidate the uncertainty quantification they rely on. The diagnostic bounds provided allow developers to estimate the risk of coverage loss without needing large amounts of labeled test data, enabling more robust deployment of LLMs in sensitive domains.