Paul Kassianik, Baturay Saglam, Huaibo Zhao, Blaine Nelson, Supriti Vijay, Aman Priyanshu, Amin Karbasi
7 min
Abstract
Multi-step LLM pipelines fail through interactions among retrieval, reasoning, and formatting steps, so prompt-only optimization can miss bottlenecks in the chain. We present FAPO (Fully Autonomous Prompt Optimization), a framework that lets Claude Code optimize an LLM pipeline inside a standardized codebase. FAPO evaluates a pipeline, inspects intermediate steps, diagnoses failures, proposes scoped changes, and validates variants repeatedly to optimize against a score function. It first tries prompt edits and, only when prompt optimization appears insufficient, changes chain structure within the permitted scope when attribution identifies a structural bottleneck. Across six benchmarks and three task models, FAPO beats the baseline GEPA in 15 of 18 model-benchmark comparisons. In 11 model-benchmark comparisons, FAPO wins with non-overlapping mean $\pm$ trial-standard-deviation ranges, and the mean FAPO-GEPA gain is +14.1 pp. In the six HoVer and IFBench comparisons where prompt-first search escalated to structural changes, FAPO wins all six with a mean gain of +33.8 pp. FAPO also improves performance on security tasks: on CTIBench-RCM, a security CVE-to-CWE task, prompt-only FAPO lifts test accuracy by +4.0 pp on GPT-5, +7.1 pp on Foundation-Sec-8B-Instruct, and +2.0 pp on Foundation-Sec-8B-Reasoning. These results position FAPO as a state-of-the-art pipeline optimization technique for both general-purpose and security-focused tasks.
Alex: So it's not a free-for-all. It's more like a contractor who can renovate your kitchen, but only within the plans you've approved.
Sam: That's a good way to put it. And the way it learns whether a change actually helped is through what they call an "eval runner." Think of it like a test at the end of a lesson. The system takes the updated pipeline and runs it against a set of test questions. If the scores improve, the change is kept. If not, it's logged and discarded.
Alex: So it's running experiments, essentially. Try a change, measure the result, keep what works.
Sam: Exactly. And there's a safeguard built into that process too. The system can look at individual examples to learn from specific mistakes, but the final judgment of whether a change is an improvement only happens by looking at overall scores across many examples. This prevents a common problem called "overfitting"—where a system gets very good at the specific test questions it's seen, but fails on anything new. Think of it like a student who memorizes last year's exam answers instead of actually learning the subject.
Alex: And all of this is recorded somewhere, so you can trace what the system did?
Sam: Yes. Every attempt—whether accepted or rejected—is saved. The researchers call this "iteration memory." If a change fails, the system logs why, so it doesn't repeat the same mistake. And because every version is saved as a separate file, an engineer can always roll back to an earlier version if something goes wrong.
Alex: Can you give a concrete example of this actually changing something meaningful in a pipeline?
Sam: There's a clear one in the paper. One of the benchmark tasks involves verifying facts across multiple documents—you have to cross-reference several sources to confirm whether a claim is true. The baseline pipeline wasn't retrieving enough information to do this reliably. Because FAPO had the authority to change the structure, it extended the pipeline by adding extra retrieval stages. It didn't just tell the AI to "try harder." It physically changed the workflow to gather more evidence before drawing a conclusion.
Alex: That's a meaningful distinction. It's the difference between coaching a runner to run faster and actually changing the route they're running.
Sam: That's a fair comparison. And the results across the paper's benchmarks reflect this. Across six different tasks, FAPO outperformed the approach of only optimizing prompts in fifteen out of eighteen cases. The largest gains came in cases where the system was permitted to escalate to structural changes—which suggests that for complex workflows, the ability to modify the architecture is where the real advantage lies.
Alex: Though I'd imagine that flexibility also makes the system less predictable? If it's choosing between different types of fixes, the outcome might vary depending on which path it takes?
Sam: The researchers noted exactly that. They observed higher variation between runs compared to simpler methods. The reason is what they call "path dependence." In one run, the system might detect a structural bottleneck early and significantly redesign the pipeline. In another run on the same task, it might stay at the prompt-editing level. The variation isn't random noise—it reflects the system genuinely exploring different solutions to the same problem.
Alex: Like two mechanics who both fix the car, but one replaces a part while the other adjusts the calibration. Different paths, same destination.
Sam: And the researchers also flagged an interesting constraint that affects some AI models specifically. Certain models share a single budget between their internal reasoning process and their final written output. If the model spends too much of that budget thinking through a problem, it runs out of space to actually write the answer—and produces an incomplete response.
Alex: So thinking too hard can cause it to fail?
Sam: In a sense. FAPO's diagnostic logs caught this pattern. For those models, the system had to explicitly manage the trade-off between how much reasoning the model does and how much room it leaves for the output. It's a constraint that a simple prompt-rewriting tool would likely miss entirely, because it wouldn't have visibility into what was happening at each step.
Alex: So the real value isn't just the performance numbers—it's that the system can adapt to the specific limitations of whichever model it's working with.
Sam: That's the key point. By treating the pipeline as a transparent, inspectable system rather than a black box, FAPO moves AI development away from trial-and-error and toward something closer to a proper engineering discipline—where you diagnose, test, and fix in a structured way. It's a meaningful step toward making these multi-step AI workflows reliable enough for real-world use.
Alex: A structured, evidence-based approach to building AI systems that actually hold up under pressure. Thanks for walking us through it, Sam, and thanks to everyone listening to ResearchPod.