Sparse Concept Channels in Frozen 3D CT Vision Encoders

As large vision-language models (VLMs) become standard for interpreting 3D medical images, their internal decision-making processes remain opaque. This study investigates how clinical findings are represented within these models, specifically asking which internal units (channels) encode specific radiological findings and whether this information can be accessed without the computational cost and risk of fine-tuning.

The authors introduce the Concept Channel Probe (CCP), a training-free method to interpret frozen 3D vision encoders. Instead of updating model weights, they treat the frozen vision embeddings as a black box. By using multi-label volume annotations, they rank embedding channels based on their selectivity for specific clinical findings. They demonstrate that a sparse subset of approximately 10 channels is sufficient to capture the information required for classification. They further validate these findings through causal ablation—showing that zeroing out these specific channels selectively degrades performance for the associated finding—and by applying the same probe to an architecturally distinct abdominal VLM.

The study reveals that radiological concepts are sparsely distributed across frozen encoder channels. The CCP-10 method (using the top 10 channels) achieves classification performance comparable to full-feature models and significantly outperforms zero-shot text prompting. When paired with a corpus-derived report template, this approach generates clinical reports with higher accuracy and better linguistic quality than existing trained models like CT-CHAT, while operating at 22 times lower latency. The results suggest that this sparse, interpretable organization is a general property of frozen medical VLMs, rather than an artifact of a single architecture.

This work provides a reproducible, lightweight framework for interpreting and utilizing frozen medical foundation models. By decoupling detection from report generation and avoiding fine-tuning, researchers can build efficient, explainable clinical decision support tools that are portable across different model backbones and anatomical domains.