Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift: A Retrospective Evaluation

Research Approach

This study evaluates a selective-prediction framework designed to assist in thyroid nodule triage using ultrasound images. The researchers developed a five-member deep ensemble based on the ConvNeXt-Tiny architecture, incorporating squeeze-and-excitation attention to improve feature representation. The framework uses member-wise vector-scaling for probability calibration and mutual information (MI) as a measure of ensemble disagreement to decide whether to provide an automated recommendation or defer the case to a radiologist.

Triage Policy and Performance

The system employs a three-tier policy: images with high ensemble disagreement or intermediate probability scores are referred for radiologist review, while only high-confidence, low-disagreement cases receive an automated No-FNA or FNA recommendation. In internal testing on the TN5000 dataset, the model achieved high discrimination (AUC-ROC 0.9395) and excellent calibration. However, when the frozen policy was applied to the external TN3K dataset, discrimination dropped (AUC-ROC 0.7870) and calibration deteriorated significantly (ECE 0.1899), demonstrating that the model's operating thresholds are sensitive to dataset shifts.

Clinical Implications

The findings suggest that while selective prediction—using uncertainty to withhold unreliable automated suggestions—is a promising strategy for clinical safety, it is not a plug-and-play solution. The significant performance drop on external data highlights that deep learning models for thyroid ultrasound require rigorous local recalibration and prospective validation before they can be safely deployed in clinical practice. The study emphasizes that automated systems should be viewed as tools to support, rather than replace, expert radiologist assessment.