Sambit Tarai, Ashish Chauhan, Elin Lundström, Johan Öfverstedt, Therese Sjöholm, Veronica Sanchez Rodriguez, Håkan Ahlström, Joel Kullberg
5 min
Abstract
Purpose: Automated medical image-based prediction of clinical outcomes, such as overall survival (OS), has great potential in improving patient prognostics and personalized treatment planning. We developed a deep regression framework using tissue-wise FDG-PET/CT projections as input, along with a temporal input representing a scalar time horizon (in days) to predict OS in patients with Non-Small Cell Lung Cancer (NSCLC). Methods: The proposed framework employed a ResNet-50 backbone to process input images and generate corresponding image embeddings. The embeddings were then combined with temporal data to produce OS probabilities as a function of time, effectively parameterizing the predictions based on time. The overall framework was developed using the U-CAN cohort (n = 556) and evaluated by comparing with a baseline method on the test set (n = 292). The baseline utilized the ResNet-50 architecture, processing only the images as input and providing OS predictions at pre-specified intervals, such as 2- or 5-year. Results: The incorporation of temporal data with image embeddings demonstrated an advantage in predicting OS, outperforming the baseline method with an improvement in AUC of 4.3%. The proposed model using clinical + IDP features achieved strong performance, and an ensemble of imaging and clinical + IDP models achieved the best overall performance (0.788), highlighting the complementary value of multimodal inputs. The proposed method also enabled risk stratification of patients into distinct categories (high vs low risk). Heat maps from the saliency analysis highlighted tumor regions as key structures for the prediction. Conclusion: Our method provided an automated framework for predicting OS as a function of time and demonstrates the potential of combining imaging and tabular data for improved survival prediction.
Alex: Like sampling clips from a patient's timeline to learn the whole story. How does it compare to other methods doctors use, like DeepHit or DeepSurv?
Sam: DeepHit breaks time into steps like calendar months, handling competing risks. DeepSurv updates the Cox method, where risk factors multiply a baseline death rate, using networks for flexibility. This method beat both on average accuracy across time points, using images or clinical data.
Alex: It beats them without clinical extras. What about training speed?
Sam: One network trains much faster than ten separate ones. It resists overfitting with varied samples. Adding tumor-specific image views boosted results most.
Alex: For risk groups, they used that area score over time?
Sam: Yes, AUSPC totals predicted survival odds over five years—higher means lower risk, like more area under a steady curve. K-means clustering split patients into high and low risk, with clear survival differences on Kaplan-Meier plots, especially by tumor stage. Ensembling images with clinical data hit the highest scores.
Alex: Limits like small subgroup sizes?
Sam: Yes, the paper cautions on that, preserving rigor. It also notes a slight underestimation of long-term survival on plots, and reliance on one hospital's data.
Alex: Why 2D views instead of full 3D?
Sam: Full 3D risks overfitting—like cramming quirks for one test instead of general rules. 2D tissue views cut complexity while keeping essentials for better generalization.
Alex: Images with clinical data gave the edge because scans catch tumor details notes might miss.
Sam: Exactly. Combined, they yielded top scores. With cautions on compute needs and single-site data, it suggests a solid tool for flexible forecasts from routine scans.
Alex: Pulling it together, this single model offers adjustable survival odds from scans, outperforming fixed-time setups and others, with tumor-focused insights and clear risk splits.
Sam: Yes. It avoids wasting censored data and captures risk changes—a meaningful step for personalized planning. The code is public for further work.
Alex: This makes survival estimates more practical for doctors. Thanks for listening to ResearchPod.