Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging | Nishanth Arun et al. | ResearchPod

Transcript: Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging

Alex: Welcome to another episode of ResearchPod.

Sam: Today we're looking at a study on something called "saliency maps." When an AI looks at a medical scan and makes a diagnosis, these maps are supposed to show *why* it reached that conclusion—highlighting the pixels that most influenced its decision, like a heat map overlaid on the image. The study's central finding is that many of these tools fail basic reliability tests, which could seriously mislead the doctors using them.

Alex: So the question is: do these heat maps actually show the AI's reasoning, or are they just showing us noise that *looks* meaningful?

Sam: Exactly. Think about a radiologist using AI to check a chest scan for a collapsed lung. They need to know the AI is actually looking at the lung—not at some irrelevant corner of the image. If the heat map is unreliable, it gives the doctor a false sense of security. They think they understand the AI's reasoning, but they don't.

Alex: So the researchers set up a framework to stress-test these tools. What did they actually check for?

Sam: Two things, mainly. First, can the map accurately point to where the abnormality is? Second—and this is the more revealing part—if you deliberately scramble the AI's internal settings, does the map change? It should. If the map stays the same even after you've essentially broken the AI's internal logic, that tells you the map was never really connected to the AI's reasoning in the first place. It's like a witness who tells the exact same story whether the crime happened or not. You can't trust that testimony.

Alex: And how did the tools hold up?

Sam: Poorly, across the board. Every single technique the researchers tested failed at least one of their criteria. What's particularly telling is that many of these sophisticated, AI-generated explanations were outperformed by something far simpler—a static "average mask." That's just a map showing where abnormalities *typically* appear across the whole dataset, with no knowledge of the specific patient or image in front of it.

Alex: Wait—so a generic template that ignores the actual scan did a better job of pointing to the problem than the AI's own explanation?

Sam: In terms of pinpointing the location, yes. And the study found that models *specifically designed* to draw boundaries around abnormalities—rather than just classify whether a disease is present or not—significantly outperformed the saliency maps as well. Which points to a deeper issue.

Alex: Which is that these saliency maps are being asked to do a job they were never built for?

Sam: Precisely. A model trained only to say "yes, disease present" or "no, disease absent" isn't the same as a model trained to say "here is exactly where the disease is." When you try to extract location information from a classification model using a saliency map, you're essentially asking it to answer a question it was never taught to answer. The map might look convincing, but the study suggests it often isn't grounded in the model's actual decision-making.

Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging

Nishanth Arun, Nathan Gaw, Praveer Singh, Ken Chang, Mehak Aggarwal, Bryan Chen, Katharina Hoebel, Sharut Gupta, Jay Patel, Mishka Gidwani, Julius Adebayo, Matthew D. Li, Jayashree Kalpathy-Cramer

Jun 20, 2026·3:38·user_upload

Saliency MapsConvolutional Neural Networks (CNNs)Model InterpretabilityLocalization UtilityModel Randomization TestTrustworthiness Criteria

Listen

4 min

Briefing

Research Question

As deep learning models become standard in radiology, clinicians require methods to interpret model decisions. Saliency maps—visual heatmaps highlighting image regions that influence a model's prediction—are frequently used for this purpose. This study investigates whether these maps are actually trustworthy for localizing clinical abnormalities, assessing their utility, sensitivity to model weights, repeatability, and reproducibility.

Approach

The researchers evaluated eight common saliency methods (including Gradient, Smoothgrad, Integrated Gradients, and XRAI) using two large, publicly available chest radiograph datasets: the SIIM-ACR Pneumothorax Segmentation dataset and the RSNA Pneumonia Detection Challenge dataset. They compared these methods against two baselines: a low-performance baseline (a simple average of all ground-truth masks) and a high-performance baseline (dedicated segmentation or detection networks like U-Net and RetinaNet). The study measured localization utility via the area under the precision-recall curve (AUPRC) and assessed robustness through model weight randomization, intra-architecture repeatability, and inter-architecture reproducibility.

Main Findings

The results indicate that saliency maps are poor substitutes for dedicated localization models. All eight tested saliency methods failed at least one of the four trustworthiness criteria. In terms of utility, most saliency maps performed no better—and often worse—than a simple average mask of all training data. Dedicated models like U-Net and RetinaNet significantly outperformed all saliency methods in localizing abnormalities. Furthermore, several saliency methods failed the model randomization test, meaning they produced similar maps even when the underlying model weights were randomized, suggesting the maps were not actually reflecting the model's learned features.

Why It Matters

Saliency maps are often used to justify clinical trust in AI, but this study suggests they may provide a false sense of security. Because these maps fail to reliably localize abnormalities and often lack sensitivity to the model's actual training, they are not suitable for high-stakes clinical decision-making. The authors strongly recommend that if the goal is to localize an abnormality, researchers should use dedicated segmentation or detection architectures rather than relying on post-hoc interpretability methods.

Key Terms and Definitions

Saliency Map — A visualization technique that highlights specific pixels or regions in an input image that contribute most to a model's prediction.
AUPRC — The area under the precision-recall curve, a metric used to evaluate performance on imbalanced datasets where positive cases are rare.
Model Weight Randomization — A stress test where a model's parameters are randomized to see if the interpretability method remains sensitive to the model's actual learned knowledge.
SSIM — The structural similarity index measure, a metric used to quantify the similarity between two images or maps.
Localization Utility — The ability of an interpretability method to accurately identify the spatial location of a clinical abnormality within an image.

Assessing the Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging: Research Paper Podcast