Vinicius Anjos de Almeida, Nícolas Henrique Borges, Leonardo Vicenzi, Helena Kociolek, Sarah Miria de Castro Rocha, Frederico Nassif Gomes, Júlia Cristina Ferreira Ribeiro, Lucas Emanuel Silva e Oliveira
5 min
This study evaluates the performance of various agentic Large Language Model (LLM) pipelines in generating treatment recommendations for breast cancer. As clinical decision support tools become more prevalent, understanding their reliability in complex oncology settings—where treatment plans involve systemic therapy, surgery, and radiotherapy—is critical. The researchers compared seven different architectures, ranging from simple single-model baselines to complex multi-agent systems with autonomous subagent spawning and fact-checking capabilities.
To evaluate these systems, the authors utilized 72 real clinical cases across all four stages of breast cancer. A key innovation is the introduction of Asymmetric Information Rubric Generation (AIRG), a workflow that allows an LLM to generate case-specific evaluation criteria by accessing privileged clinical data (the actual decisions made by oncologists) that the evaluated models do not see. This approach provides a grounded, expert-informed benchmark without the high costs typically associated with manual rubric creation. The models were tested on their ability to provide accurate, evidence-based recommendations, and their outputs were scored based on their adherence to these generated rubrics.
The best-performing configuration, Claude Opus 4.8 using a 'Divide and Conquer' pipeline with autonomous subagents, achieved a global score of 0.594. The study found that adding tools and increasing agent autonomy had inconsistent effects, sometimes improving performance but occasionally degrading it. A detailed error analysis by an oncologist revealed significant, clinically relevant failures, including the fabrication of references, citation errors, outdated medical claims, and a tendency toward overconfidence. These results indicate that while agentic systems show promise, they are not yet ready for autonomous deployment in clinical practice.
Large language models (LLMs) are increasingly being explored for clinical decision support, but their reliability in complex oncology treatment planning remains unclear. We evaluated agentic LLM systems for breast cancer treatment recommendation generation using 72 real clinical cases across stages I to IV and 1,147 case-specific rubrics generated through Asymmetric Information Rubric Generation (AIRG), in which the rubric generator had access to real clinical decisions unavailable to the evaluated models. Seven pipelines were compared, including single-LLM baselines, tool-augmented systems, and multi-agent architectures with fact checking and autonomous subagent spawning. The best-performing configuration, Claude Opus 4.8 with the D&C+SA pipeline, achieved a global score of 0.594 $\pm$ 0.025. Tool use and increased agent autonomy had mixed effects, improving performance in some settings but degrading it in others. Performance varied by clinical domain and disease stage, and oncologist-led error analysis revealed persistent clinically relevant failures, including incorrect or missing recommendations, flawed justifications, citation errors, outdated claims, and overconfidence. These findings suggest that agentic LLM systems can generate clinically relevant breast cancer recommendations, but remain insufficient for unsupervised clinical use.
Sam: So it can go look things up in real time. Does that help it fact-check its own work?
Alex: Not reliably. The study shows that even with access to current data, the model can still misinterpret what it finds. And in some cases, the AI produced fabricated citations—made-up references that looked completely legitimate to anyone who didn't go and check. A dedicated fact-checking agent was part of some setups, and even that didn't fully solve the problem.
Sam: So even with a built-in checker, it still invents sources?
Alex: That's what the study found. The errors don't look like errors. They look like confident, well-formatted medical reasoning. That's precisely what makes them dangerous in a clinical setting.
Sam: Which is why the human oncologist has to be the final judge. The AI is a starting point, not a conclusion.
Alex: Exactly. And the study found this was especially true for the most complex cases. Performance was lowest for Stage III cancers—cases where there are many possible treatment paths and the clinical picture is less clear-cut. The more diverse the options, the harder it is for the AI to reliably identify the right one.
Sam: That makes sense. It's easy to get the straightforward cases right. The hard ones are where the stakes are highest.
Alex: And that's the tension at the heart of this research. These systems can generate relevant, useful information. They're not useless. But the gap between "useful starting point" and "reliable enough to act on without expert review" is still significant. The researchers are clear that these tools are not ready for unsupervised clinical use.
Sam: So what's the practical takeaway? Where does this leave AI in oncology?
Alex: The researchers frame it as a foundation, not a finish line. The AIRG evaluation method itself is a meaningful contribution—it gives the field a more rigorous way to measure progress. And the study points toward what needs to improve: not just the AI's access to information, but its ability to reason carefully through that information and know when it's uncertain.
Sam: It's almost like the difference between a student who can look everything up and a student who actually understands the material. Access to information isn't the same as judgment.
Alex: That's a fair way to put it. And in medicine, judgment is everything. These systems still need a trained clinician to review their outputs, catch their errors, and make the final call. The research suggests that's not a temporary limitation we're about to solve—it's a structural reality that should shape how these tools are deployed.
Sam: It's a useful reminder that passing a medical exam and practicing medicine are two very different things.
Alex: They are. And this study is a careful attempt to measure exactly that gap. Thanks for listening to ResearchPod.