Closing the Calibration Gap in Semantic Caching

Semantic caching reduces LLM inference costs by serving cached responses for semantically similar queries. Current industry standards evaluate these systems using Precision-Recall AUC (PR-AUC), which measures how well a model ranks candidates. However, this paper demonstrates that PR-AUC is fundamentally flawed for this application because it is agnostic to the absolute score thresholds required for real-time cache decisions. A model can achieve high PR-AUC by ranking correctly while remaining completely unusable at any practical cache-hit threshold.

To bridge the gap between offline ranking and online performance, the authors introduce two new metrics:

• P-CHR AUC (Precision–Cache Hit Ratio AUC): Measures the trade-off between precision and cache utilization as the decision threshold varies, directly reflecting how the system behaves in production.
• Calibration Retention Rate (CRR): Quantifies the proportion of offline ranking quality that survives the transition to deployment, highlighting how much performance is lost due to score miscalibration.

The authors decompose the performance difference between offline and online settings into two parts: a structural gap, which is irreducible and determined by the dataset's positive rate, and a calibration gap, which is a recoverable loss caused by poor score distribution. Experiments across 9 retrievers and 10 rerankers show that models with the highest PR-AUC often exhibit the worst deployment performance. Specifically, Binary Cross-Entropy (BCE) training causes severe score compression, while Multiple Negatives Ranking Loss (MNRL) improves calibration but does not eliminate the gap. The study finds that this calibration issue is a function of the training objective rather than the scale of the training data.