Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference

Large language model (LLM) cascades—where a small, cheap model attempts a query and escalates to a larger, expensive model if uncertain—are standard for managing inference costs. However, existing cascades rely on heuristic confidence thresholds that are notoriously miscalibrated and require extensive per-domain tuning. This paper asks: can we replace these heuristics with a mathematically grounded framework that provides formal, distribution-free accuracy guarantees while maintaining cost efficiency?

The authors introduce Conformal Cascade (CC), which leverages split conformal prediction to manage deferral. Instead of using raw confidence scores, CC constructs a prediction set at each tier based on the sampling frequency of answers. The cascade accepts an answer if the prediction set collapses to a single candidate (or a set of size κ) and defers otherwise. This approach turns the user-specified error budget (α) into a concrete, calibrated threshold. The framework requires no model training, works with black-box API access, and allows for a closed-form calculation of expected inference costs.

CC provides a formal cascade-level coverage bound, ensuring that the probability of the accepted answer being correct is at least 1 - Kα (or 1 - α under specific selection-preservation conditions). In empirical evaluations across 18 multiple-choice benchmarks and four model families, CC consistently outperformed the strongest calibration-tuned heuristic baselines. The method showed the most significant gains on reasoning-heavy tasks where traditional majority-vote heuristics often fail. On simpler tasks, the cascade successfully routed the vast majority of queries to the smaller, cheaper model without sacrificing accuracy.

This work bridges the gap between theoretical conformal prediction and practical LLM deployment. By replacing arbitrary confidence thresholds with a statistically valid decision rule, practitioners can now set a target error rate and have the cascade automatically determine the optimal deferral thresholds. This provides a reliable, interpretable, and cost-effective way to scale LLM applications without the need for expensive, domain-specific model retraining.