ConfTriage: A Calibration-Aware LLM Triage Framework for Pulmonary Nodule Malignancy with Selective Specialist Deferral | Md Rabiul Islam et al. | ResearchPod

Transcript: ConfTriage: A Calibration-Aware LLM Triage Framework for Pulmonary Nodule Malignancy with Selective Specialist Deferral

Alex: Welcome to another episode of ResearchPod. Today we're looking at a study that asks a genuinely interesting question: can an artificial intelligence system trained on general text help doctors sort through thousands of lung scan results — without needing to be specially trained for every new medical task?

Sam: So the paper is asking whether we can take a general-purpose AI and plug it into a medical workflow, even though it was never trained on medical images?

Alex: Exactly. The central puzzle is whether we can skip the intensive, expensive process of training a model on raw scan images — by feeding it structured written descriptions instead.

Sam: That's a meaningful distinction. What's the proposed solution?

Alex: The authors introduce a system called ConfTriage. It uses language as its primary input, and it has a built-in safety mechanism: when the system isn't confident enough about a case, it automatically hands that case off to a specialist model rather than guessing.

Sam: Walk me through how that actually works. Where does the data come from?

Alex: A radiologist examines a CT scan and records specific attributes — the size of a nodule, its texture, its position in the lung. The system takes those structured notes and converts them into a plain English sentence using a fixed template. Every patient gets the same sentence structure, just with different numbers plugged in.

Sam: So the AI never actually looks at the scan image itself — it reads a written summary of what the radiologist observed.

Alex: That's right. And that consistency matters. Because the sentence format never changes, the language model can process cases reliably without being confused by variations in how different doctors phrase their notes.

Sam: And then what does the language model actually do with that sentence?

Alex: It reads the description and produces a confidence score — essentially, it says how certain it is that a nodule is malignant. But here's the issue: raw confidence scores from AI models are often poorly calibrated. If the model says it's ninety percent sure, it might only be correct seventy percent of the time.

Sam: So the number it produces isn't automatically trustworthy.

Alex: Exactly. To fix that, the system applies a mathematical adjustment called Platt scaling. Think of it like checking whether a weather forecaster's predictions actually match reality — if they say seventy percent chance of rain, does it actually rain seventy percent of those times? Platt scaling corrects the model's confidence so that when it says it's ninety percent sure, that figure reflects its true accuracy.

ConfTriage: A Calibration-Aware LLM Triage Framework for Pulmonary Nodule Malignancy with Selective Specialist Deferral

Md Rabiul Islam, Samir Abdaljalil, Erchin Serpedin, Hasan Kurban

Aug 11, 2026·8:08·arxiv·Original Paper

ConfTriageConfidence CalibrationSelective Specialist DeferralPulmonary Nodule Malignancy PredictionPlatt ScalingSelective Prediction

Listen

8 min

Briefing

Research Question

Pulmonary nodule malignancy prediction is traditionally performed by specialized deep learning models that require extensive image-level training and are often opaque to clinicians. This paper investigates whether a generalist Large Language Model (LLM), when provided with a structured natural-language description of nodule attributes, can serve as a reliable, calibrated triage layer to reduce the workload on specialist models.

Approach

The authors introduce ConfTriage, a framework built on three pillars: using language as the primary modality, applying Platt scaling to calibrate the LLM's verbalized confidence, and utilizing a specialist deep learning backstop (Certain-Net) for cases where the LLM lacks sufficient confidence. The researchers conducted a seven-way input ablation study across five frontier LLMs (including GPT-4o-mini, Claude-3.5-Haiku, and Gemini-3.1-Flash-Lite) on the LIDC-IDRI benchmark. They also provided theoretical proofs, including a finite-sample combined-error bound and an oracle inequality that links the LLM's L1 calibration error to the optimality of the deferral policy.

Main Findings

The study demonstrates that natural-language descriptions are the primary driver of diagnostic signal, while low-level image statistics are essentially diagnostically vacuous. ConfTriage achieved an F1 score of 88.22% and an AUC of 0.92, successfully resolving 76.5% of cases using zero-shot LLM inference alone. By calibrating the LLM's confidence, the framework provides a formal, defensible mechanism for deciding when to trust the LLM and when to defer to a specialist, effectively bridging the gap between generalist AI and specialized medical decision-support systems.

Why It Matters

This research provides a practical, interpretable, and scalable pathway for integrating generalist LLMs into clinical workflows. By demonstrating that structured text is sufficient for high-performance triage, the authors offer a method that avoids the need for resource-intensive, end-to-end image training for every diagnostic task, while simultaneously providing the safety guarantees necessary for clinical deployment.

Key Terms and Definitions

ConfTriage — A calibration-aware triage framework that uses LLMs to classify pulmonary nodules and selectively defers uncertain cases to a specialist deep learning model.
Platt Scaling — A method used to transform the LLM's raw verbalized confidence scores into well-calibrated probabilities.
Specialist DL Backstop — A secondary, specialized deep learning model (Certain-Net) that processes cases the LLM is not sufficiently confident to classify.
LIDC-IDRI — A standard public benchmark dataset containing thoracic CT scans and annotated pulmonary nodule characteristics used for evaluating malignancy prediction.
Diagnostic Vacuity — A condition where an input modality (such as raw image statistics) provides no meaningful signal for the classification task.

Abstract

Pulmonary nodule malignancy prediction typically depends on image-trained specialist deep learning (DL) models that require substantial annotated imaging data and task-specific training. We investigate whether a generalist large language model (LLM), reading only a faithful natural-language rendering of standard nodule attributes, can serve as a calibrated triage layer. We propose ConfTriage, a confidence-calibrated method built on three pillars: language as the modality, calibration as the safety mechanism, and a selective specialist DL backstop for low-confidence cases. We prove two guarantees: a finite-sample combined-error bound yielding an explicit per-threshold operational certificate, and an oracle inequality showing that excess risk over the Bayes-optimal deferral classifier is controlled by the L1 calibration error of the LLM probability. A controlled seven-way input ablation across five frontier LLMs on LIDC-IDRI shows that natural-language descriptions dominate the diagnostic signal, while low-level image statistics are essentially diagnostically vacuous. ConfTriage achieved an F1 score of 88.22% and an AUC of 0.92, resolving 76.5% of cases using zero-shot LLM inference alone and referring only uncertain cases to the specialist DL backstop. These results demonstrate that clinically meaningful diagnostic information can be captured through structured radiological descriptions and leveraged by calibrated LLMs for selective referral. The framework suggests a practical pathway for combining generalist LLM prediction with specialist AI models in medical decision-support systems. Source code is publicly available at https://github.com/rabiul-ai/ConfTriage.

ConfTriage: A Calibration-Aware LLM Triage Framework for Pulmonary Nodule Malignancy with Selective Specialist Deferral: Research Paper Podcast