Sayeed Shafayet Chowdhury, Karen D’Souza, V. Siva Kakumani, Snehasis Mukhopadhyay, Shiaofen Fang, Rodney J. Schlosser, Daniel M. Beswick, Jeremiah A. Alt, Jess C. Mace, Zachary M. Soler, Timothy L. Smith, Vijay R. Ramakrishnan
11 min
Abstract
Artificial intelligence (AI) has increasingly transformed medical prognostics by enabling rapid and accurate analysis across imaging and pathology. However, the investigation of machine learning predictions applied to prospectively collected, standardized data from observational clinical intervention trials remains underexplored, despite its potential to reduce costs and improve patient outcomes. Chronic rhinosinusitis (CRS), a persistent inflammatory disease of the paranasal sinuses lasting more than three months, imposes a substantial burden on quality of life (QoL) and societal cost. Although many patients respond to medical therapy, others with refractory symptoms often pursue surgical intervention. Surgical decision-making in CRS is complex, as it must weigh known procedural risks against uncertain individualized outcomes. In this study, we evaluated supervised machine learning models for predicting surgical benefit in CRS, using the Sino-Nasal Outcome Test-22 (SNOT-22) as the primary patient-reported outcome. Our prospectively collected cohort from an observational intervention trial comprised patients who all underwent surgery; we investigated whether models trained only on preoperative data could identify patients who might not have been recommended surgery prior to the procedure. Across multiple algorithms, including an ensemble approach, our best model achieved approximately 85% classification accuracy, providing accurate and interpretable predictions of surgical candidacy. Moreover, on a held-out set of 30 cases spanning mixed difficulty, our model achieved 80% accuracy, exceeding the average prediction accuracy of expert clinicians (75.6%), demonstrating its potential to augment clinical decision-making and support personalized CRS care.
Alex: Layers of math connections... okay, so it's not just adding up numbers, but finding hidden links—like how bad symptoms plus certain scans signal low odds. And they combined a few of these computers together somehow?
Sam: Yes—they tested several, like ones that draw straight lines through data clouds or build decision trees like family choice charts. Then they grouped the top ones into a team: each makes its own yes-or-no call, and the final answer is whatever most agree on. This ensemble averaged out individual mistakes, landing at 86 percent accuracy on held-out patient cases. That's a clear edge over six expert nose surgeons, who averaged around 76 percent in a side-by-side test.
Alex: So the computer team beat the doctors by a noticeable margin. But why would grouping models help more than one alone?
Sam: Single models can miss angles; one might over-focus on symptoms while ignoring scans. By voting, strengths balance weaknesses—like asking a few friends for advice instead of one. The study suggests this setup, using only routine clinic notes from six sites, could help doctors counsel patients more reliably, spotting likely non-improvers to discuss alternatives first.
Alex: That points to a practical tool—not replacing doctors, but sharpening their calls with data patterns humans might overlook. So if it's sharpening decisions with patterns from routine data, what exactly went into those models—like, how did they turn messy clinic notes into something the computers could use?
Sam: They started with patient records from two big studies across multiple centers, focusing only on details from before surgery. Things like age, symptom scores, scan results, allergies, smoking history, and other health conditions got cleaned up—missing bits dropped since they were rare, and categories like race or insurance turned into simple numbers doctors agreed made sense, like coding male as 0 and female as 1. This gave 31 solid pieces of info per patient, all structured and ready, split into training and testing groups that kept the same mix of winners and non-winners.
Alex: Got it—turning words and gaps into numbers keeps it reliable across patients. And the MLP you mentioned, with its layers... does that mean it weighs some clues more than others automatically?
Sam: Yes—the first layer might spot basics like high symptom scores or bad scans, passing a summary forward. The next layer mixes in risks like age or asthma, adjusting strengths to find combinations that predict success, ending with a yes-or-no score. It learns these weights by trial and error on training data, tweaking to match real outcomes while a built-in brake stops it from memorizing quirks too closely.
Alex: Huh, so layers let it handle real-life overlaps, not just one factor at a time. How does the team-up with other models make it even steadier?
Sam: Each model sees the data a bit differently—one might prioritize scans, another symptoms—but when they vote, the group answer smooths out solo errors. The study used majority rule from the best ones, like MLP and a tree-based setup, hitting steady predictions on new patients. This ensemble acts as clinical decision support, giving doctors a quick, data-backed nudge on who might skip surgery risks.
Alex: Right—and since it's from routine checks at six sites, it could fit right into office workflows. If it's a meaningful aid from routine data, what tipped the scales for that layered model over the others—like, why not just stick with a simpler one?
Sam: The data clues weren't split neatly by straight lines, like trying to draw borders around mixed-up marbles with a ruler—it needs curves or branches to sort properly. Looking at charts of the main patterns in patient info showed lumps that simple rules missed, so bendy tools like layered math or decision trees worked better. They picked the layered one because it nailed the hard part: spotting folks unlikely to improve.
Alex: So the layers caught tricky mixes that straight fits overlooked. And they tweaked its setup—how did that fine-tune it without going overboard?
Sam: They tested layer counts and sizes on practice data, finding one middle layer with plenty of connections gave steady results without chasing noise. Brakes like weighting rarer cases heavier, a smoothness penalty, and stopping early kept it general.
Alex: Huh, so tuning avoided too much wiggle on quirks. How did this stack up against the actual surgeons in the test?
Sam: On 30 tricky held-back cases, six nose specialists guessed right about 76 percent overall. The layered model edged ahead at 85 percent, and the voting team hit 86—the first time such routine-data computers topped experts across sites. It suggests AI as a steady sidekick, blending data patterns with doctor know-how for tougher calls.
Alex: Right—that combo could cut unnecessary surgeries without sidelining judgment. A notable step for real clinics. But to trust it in clinics, doctors need to know what the model bases its calls on, right? How did they check which patient details mattered most?
Sam: They tested by scrambling one detail at a time—like mixing up ages while keeping everything else steady—and measured how much the predictions worsened. The biggest drops came from baseline symptom scores, age, scan severity, allergies, and past surgery, matching what experts expect: bad symptoms and disease signs drive the need for knives.
Alex: So symptoms lead, then body scans and risks... makes sense as a sanity check. And for each patient, did it show exactly how those clues tipped the scale?
Sam: Yes—they used a method that splits credit for the final call across all clues, like dividing a team's win among players based on every lineup combo. High symptoms or scans push toward surgery, low ones away, with income or polyps adding nudges case-by-case. It's called SHAP, from game theory ideas, and it gives doctors clear reasons per patient, building trust.
Alex: Huh—that transparency could let a surgeon say, 'Your scores suggest low odds because of X and Y.' How did it hold up against other AI, like those chatbots everyone uses?
Sam: They tested a large language model—ChatGPT—fed the same patient details via prompts for yes-or-no calls. It hit just 57 percent accuracy on the 30 cases, while the layered model reached 80 percent. The study suggests specialized training on clinic data beats general chat tools for this narrow task.
Alex: Under-calling... so it missed likely winners. And experts—did grouping them like the models help?
Sam: Six doctors averaged 76 percent; even their majority vote stayed around there, as mistakes overlapped on tough cases. Both humans and model struggled most on ambiguous ones—showing errors cluster where clues conflict. This points to AI as a sidekick for those gray areas, not a solo act.
Alex: Right—shared blind spots mean teaming data smarts with doctor insight fits best. But to make this a real tool, we have to talk limits—what held it back from being even sharper?
Sam: Fair point. The patient group was moderate-sized, around 500 cases, which caps how finely the models can learn rare patterns without overfitting. Non-responders made up a smaller share, about one in three or four, and proved noisier to predict. They eased this by giving those cases extra weight in training, but bigger, more varied data would help more. The six-month check also simplified outcomes to yes-or-no, missing later gains some might see.
Alex: So fewer tough cases and a short follow-up window... that tracks with why non-responders tripped things up. Still, even with that, how does it play out for actual patient talks?
Sam: Clinically, that level with clear breakdowns of why—like symptoms and scans leading, risks tweaking—offers solid ground for counseling. Paired with surgeon smarts, it spots skips for meds or watches, potentially trimming needless procedures where gains stay small.
Alex: Huh—a nudge toward better matches without solo calls. Looking ahead, what refines it next?
Sam: Future steps include growing data across sites with privacy tricks, boosting rare cases via smart copies or tweaks that stay realistic, and testing on fresh groups to prove it holds up. They'd also track fairness on factors like income that might skew access, eye longer symptom paths, and blend with doctor loops for ongoing tweaks. Overall, this pipeline marks a notable, interpretable step in using routine data to aid sinus surgery choices—balancing strengths against clear hurdles like sample limits.
Alex: That's a grounded take—teamwork between data and docs, with room to build. Thanks for breaking it down, Sam.
Sam: My pleasure, Alex.