Sai Karthik Kosuri, Ankita Shashikant Bhosale, Michael Glick, Alonso Carrasco-Labra, Chris Callison-Burch
6 min
Systematic reviews are essential for clinical guidelines but are hindered by the labor-intensive, protocol-heavy nature of data extraction. EviStreams is an open-source, no-code web platform designed to assist these reviews by integrating AI into the established, auditable workflows required by medical research. The system emphasizes human-in-the-loop control, allowing domain experts to define structured fields, inspect AI-extracted values alongside their source text, and perform blinded dual reviews with adjudication.
EviStreams operates through three key stages: program design, where a methodologist approves an extraction plan; field specification, where typed definitions are calibrated using pilot papers; and prediction, where AI-generated data is verified by two independent reviewers. Unlike systems that focus solely on model performance, EviStreams provides a control surface for experts to refine hints, rules, and examples for each field. The system also employs an anchor-column strategy for table extraction to prevent row and column drift, ensuring that data is correctly mapped to specific study arms.
The researchers evaluated EviStreams across four clinical corpora using three frontier LLM families. They found that the choice of model had a negligible impact on overall performance, with extraction quality remaining robust across different providers. Instead, the most significant driver of quality was the field specification; richer, more detailed definitions consistently improved performance, particularly for complex fields requiring disambiguation. The study also analyzed 1,003 discrepancies, finding that two-thirds were attributable to methodological ambiguities or ground-truth conventions rather than actual AI model failures.
By demonstrating that field specification is the most effective lever for improving extraction quality, EviStreams shifts the focus from model selection to expert-driven task design. This approach aligns with the rigorous evidentiary standards of systematic reviews, providing a transparent, auditable, and reproducible workflow that reduces the expert-labor bottleneck while maintaining high accuracy.
Systematic reviews underpin clinical guidelines, yet their data-extraction step is a major expert-labor bottleneck bound by a protocolized workflow: two reviewers extract each study independently, an adjudicator resolves disagreements, and the team keeps an auditable record of how every value was produced. Large language models can assist with extraction, but that assistance must fit established review protocols and preserve reproducibility. We present EviStreams, a live, open-source, no-code web platform that puts review teams in control of AI-assisted extraction at three key stages: program design (a structured decomposition approved before any code runs), field specification (typed field definitions calibrated from a pilot), and extracted predictions (reviewer-blinded dual review with adjudication). Working through a form builder, a domain expert defines typed fields rather than prompts, runs extraction over uploaded PDFs, inspects every value alongside the supporting passage it came from, and resolves a reviewer-blinded dual review into an auditable consensus export. An evaluation across four clinical corpora and three frontier model families, released with the system, shows that extraction quality is shaped far more by the field specification than by the choice of model. EviStreams is live at https://evistreams.com/demo and released under Apache-2.0.
Sam: [precise] Right. Every extracted value stays linked to the verbatim quote and PDF location it came from, so the dataset is auditable rather than just editable. That source grounding isn't decorative — it's what lets a methodologist verify whether a value is actually supported by the paper or just plausible. [[RP_SECTION:table-decomposition-strategy|Table Decomposition Strategy]]
Alex: [slightly faster pace] What about tables? That's usually where extraction gets brittle.
Sam: [building the case] They handle tables by decomposition. Instead of asking a model to read a dense table positionally, they first extract a small set of anchor columns that identify each row — a drug name and dose, say — then fan out one call per row to fill the rest. Fixing row identity first makes it less likely the model mixes up which arm or follow-up point it's filling.
Alex: [curious] Is that actually a major source of gain, or just a cleaner implementation?
Sam: [careful] The paper is fairly restrained there. Decomposition gives governable structure, but the reported accuracy gains are small and mixed in direction. The stronger claim isn't that decomposition alone boosts F1 across the board — it's that it makes the workflow more controllable and less fragile, which is what a review team actually needs. [[RP_SECTION:evaluation-and-methodology|Evaluation and Methodology]]
Alex: [reflective] Then the evaluation really has to separate model effects from specification effects. How did they test that?
Sam: [steady] They fixed two of the three design levers at a time — model, pipeline, and field specification — and varied the third, testing this across four clinical corpora and several frontier model families. The corpora came from completed reviews in oral cancer, antibiotic prophylaxis, periodontitis, and ibuprofen, so the reference data already followed the same dual-review protocol the system is meant to support. Scoring used a field-typed harness with a global assignment step for record alignment, so missing or spurious arms were penalized sensibly.
Alex: [genuine confusion] I'm not sure I follow why the specification would dominate so strongly. If the model's already strong, shouldn't it recover the field from the paper anyway?
Sam: [patient, clarifying] Not when the extraction target is underspecified or convention-heavy. A model can read the paper, but it still has to infer which number counts, which arm is the analyzed one, whether a table row is an outcome or a follow-up point, and where the value is localized in the text. A richer specification narrows those ambiguities by telling the system what to look for, what to ignore, and what shape the answer should take. That's why the biggest gains show up when a field needs disambiguation or source localization, not when it's already obvious.
Alex: [analytical] So the headline isn't "better prompts help" — it's that the review protocol itself becomes the main performance lever.
Sam: [measured] That's the right reading. The ablation showing richer specifications usually help is the load-bearing evidence; model choice and pipeline decomposition matter less and inconsistently, so they're scaffolding around that main result. The error analysis is more of a secondary layer — many discrepancies turned out to be methodological rather than genuine model failures, and the real errors clustered in dense, multi-arm, multi-table reporting. That tells you where the system is still brittle. [[RP_SECTION:generalization-and-limitations|Generalization and Limitations]]
Alex: [slower, probing] And the limitation is that they calibrated those specifications on pilot papers drawn from the same reviews, which is a fairly favorable setup.
Sam: [brief pause, then honest] Yes, and that's the main constraint on how far the result generalizes. The gain from specification tuning may be overstated relative to a cold-start review where no pilot data exist yet — they say that directly. So it's strongest as evidence for review-specific calibration, not as a universal recipe that transfers untouched to a new corpus.
Alex: [reflective, quieter] That still seems useful, though. It suggests scarce expert time should go into the form itself, not chasing a different model release.
Sam: [settling the point] Exactly. The practical implication is a workflow where systematic-review protocols behave more like living documents — the team uploads new PDFs, refines the specification as edge cases appear, and keeps the audit trail intact while the AI handles the first pass. Human effort shifts toward adjudication and synthesis, which is probably where it belonged all along.
Alex: If you want the figures and the calibration details we skipped, you can generate a deep dive of this paper — the paper has the rest either way.
Sam: Thanks for listening.