What Predicts Correctness in Text-to-SQL? A Selective-Prediction Study

This study investigates how to reliably predict whether an AI-generated SQL query is correct—defined as producing the same result as a human-written reference query. The author evaluates various signals for 'selective prediction,' which allows a system to decide whether to provide an answer or abstain when confidence is low. The study compares black-box statistical signals (such as self-consistency, structural agreement, and executability) against logic-aware verification signals (using LLMs as judges) across the BIRD and Spider benchmarks. The author also explores whether smaller, fine-tuned verifiers can match the performance of large, frozen models.

The study identifies a performance ceiling for black-box statistical signals, which consistently yield an AUROC between 0.61 and 0.68. Notably, query executability—whether a query runs without error—is essentially uninformative, as incorrect queries frequently execute successfully. White-box log-probabilities also fail to exceed this ceiling, suggesting that simple statistical agreement is insufficient for complex multi-table SQL tasks.

Verification-based signals, where an LLM is prompted to evaluate the logic of the generated SQL, consistently outperform the black-box ceiling. A two-provider ensemble of judges (e.g., GPT-4o and Claude) achieves the highest performance (0.82 AUROC) and superior calibration, allowing for effective abstention policies that statistical methods cannot support. While fine-tuned verifiers achieve high accuracy on databases seen during training, they suffer from a significant performance drop when applied to unseen schemas. The author concludes that while fine-tuning is effective for fixed, in-domain deployments, universal correctness prediction currently requires large, frozen reasoning models.