Multi-step LLM pipelines fail through interactions among retrieval, reasoning, and formatting steps, so prompt-only optimization can miss bottlenecks in the chain. We present FAPO (Fully Autonomous Prompt Optimization), a framework that lets Claude Code optimize an LLM pipeline inside a standardized codebase. FAPO evaluates a pipeline, inspects intermediate steps, diagnoses failures, proposes scoped changes, and validates variants repeatedly to optimize against a score function. It first tries prompt edits and, only when prompt optimization appears insufficient, changes chain structure within the permitted scope when attribution identifies a structural bottleneck. Across six benchmarks and three task models, FAPO beats the baseline GEPA in 15 of 18 model-benchmark comparisons. In 11 model-benchmark comparisons, FAPO wins with non-overlapping mean $\pm$ trial-standard-deviation ranges, and the mean FAPO-GEPA gain is +14.1 pp. In the six HoVer and IFBench comparisons where prompt-first search escalated to structural changes, FAPO wins all six with a mean gain of +33.8 pp. FAPO also improves performance on security tasks: on CTIBench-RCM, a security CVE-to-CWE task, prompt-only FAPO lifts test accuracy by +4.0 pp on GPT-5, +7.1 pp on Foundation-Sec-8B-Instruct, and +2.0 pp on Foundation-Sec-8B-Reasoning. These results position FAPO as a state-of-the-art pipeline optimization technique for both general-purpose and security-focused tasks.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a new approach to fixing complex AI systems—one that goes beyond simply rewriting instructions and actually digs into the structure of how those systems are built. The research introduces a framework called FAPO, which stands for Fully Autonomous Prompt Optimization.
Alex: What's the core problem it's trying to solve?
Sam: To understand that, think about how modern AI systems actually work. They're not just a single AI answering a question—they're more like an assembly line. One step searches for relevant information, another reasons about it, a third formats the final answer. Each step hands off to the next. These are called pipelines.
Alex: And when something goes wrong in that assembly line, it's not always obvious which station is the problem?
Sam: Exactly. And most current tools treat the pipeline as a fixed structure and only try to rewrite the instructions—called "prompts"—given to the AI at each step. But if the underlying logic is flawed, changing the wording of an instruction won't help. It's like a car that won't start. You could keep trying different keys, but if the fuel line is broken, no key is going to fix it. FAPO acts like a mechanic who actually opens the hood and checks what's broken before deciding how to fix it.
Alex: So it's about finding the actual source of the problem, rather than hoping a different prompt will patch over a structural flaw.
Sam: Precisely. The system uses a process the researchers call "failure attribution." It runs the pipeline, collects the output from every intermediate step, and uses an AI agent to identify exactly where things go wrong—whether the failure happened during the search for evidence, during the reasoning phase, or somewhere else. By pinpointing the specific point of failure, the system can make a more informed decision about what kind of fix is actually needed.
Alex: So it's not just diagnosing that something is broken—it's diagnosing *where* it's broken, and then choosing the right tool for that specific problem?
Sam: That's the core mechanism. It follows a strict escalation path. It tries the simplest fix first—adjusting the wording of a prompt. If the diagnostic data shows the problem is deeper than that, it moves up to modifying the actual structure of the pipeline: changing how the steps connect, or adding new steps entirely. And all of this happens within boundaries set by the developer.
Alex: Which brings up the obvious question—if it's autonomously rewriting code, how do you stop it from doing something it shouldn't?
Sam: The researchers built in what they call a "scope contract"—essentially a rulebook the system must follow. Before any change is applied, a separate reviewer agent checks the proposal against strict rules. If a proposed change could accidentally leak private data, break the formatting the system depends on, or step outside the allowed boundaries, it's rejected. The system has to try something else. It has the power to change the architecture, but it's always working within human-defined limits.
Alex: So it's not a free-for-all. It's more like a contractor who can renovate your kitchen, but only within the plans you've approved.
Sam: That's a good way to put it. And the way it learns whether a change actually helped is through what they call an "eval runner." Think of it like a test at the end of a lesson. The system takes the updated pipeline and runs it against a set of test questions. If the scores improve, the change is kept. If not, it's logged and discarded.
Alex: So it's running experiments, essentially. Try a change, measure the result, keep what works.
Sam: Exactly. And there's a safeguard built into that process too. The system can look at individual examples to learn from specific mistakes, but the final judgment of whether a change is an improvement only happens by looking at overall scores across many examples. This prevents a common problem called "overfitting"—where a system gets very good at the specific test questions it's seen, but fails on anything new. Think of it like a student who memorizes last year's exam answers instead of actually learning the subject.
Alex: And all of this is recorded somewhere, so you can trace what the system did?
Sam: Yes. Every attempt—whether accepted or rejected—is saved. The researchers call this "iteration memory." If a change fails, the system logs why, so it doesn't repeat the same mistake. And because every version is saved as a separate file, an engineer can always roll back to an earlier version if something goes wrong.
Alex: Can you give a concrete example of this actually changing something meaningful in a pipeline?
Sam: There's a clear one in the paper. One of the benchmark tasks involves verifying facts across multiple documents—you have to cross-reference several sources to confirm whether a claim is true. The baseline pipeline wasn't retrieving enough information to do this reliably. Because FAPO had the authority to change the structure, it extended the pipeline by adding extra retrieval stages. It didn't just tell the AI to "try harder." It physically changed the workflow to gather more evidence before drawing a conclusion.
Alex: That's a meaningful distinction. It's the difference between coaching a runner to run faster and actually changing the route they're running.
Sam: That's a fair comparison. And the results across the paper's benchmarks reflect this. Across six different tasks, FAPO outperformed the approach of only optimizing prompts in fifteen out of eighteen cases. The largest gains came in cases where the system was permitted to escalate to structural changes—which suggests that for complex workflows, the ability to modify the architecture is where the real advantage lies.
Alex: Though I'd imagine that flexibility also makes the system less predictable? If it's choosing between different types of fixes, the outcome might vary depending on which path it takes?
Sam: The researchers noted exactly that. They observed higher variation between runs compared to simpler methods. The reason is what they call "path dependence." In one run, the system might detect a structural bottleneck early and significantly redesign the pipeline. In another run on the same task, it might stay at the prompt-editing level. The variation isn't random noise—it reflects the system genuinely exploring different solutions to the same problem.
Alex: Like two mechanics who both fix the car, but one replaces a part while the other adjusts the calibration. Different paths, same destination.
Sam: And the researchers also flagged an interesting constraint that affects some AI models specifically. Certain models share a single budget between their internal reasoning process and their final written output. If the model spends too much of that budget thinking through a problem, it runs out of space to actually write the answer—and produces an incomplete response.
Alex: So thinking too hard can cause it to fail?
Sam: In a sense. FAPO's diagnostic logs caught this pattern. For those models, the system had to explicitly manage the trade-off between how much reasoning the model does and how much room it leaves for the output. It's a constraint that a simple prompt-rewriting tool would likely miss entirely, because it wouldn't have visibility into what was happening at each step.
Alex: So the real value isn't just the performance numbers—it's that the system can adapt to the specific limitations of whichever model it's working with.
Sam: That's the key point. By treating the pipeline as a transparent, inspectable system rather than a black box, FAPO moves AI development away from trial-and-error and toward something closer to a proper engineering discipline—where you diagnose, test, and fix in a structured way. It's a meaningful step toward making these multi-step AI workflows reliable enough for real-world use.
Alex: A structured, evidence-based approach to building AI systems that actually hold up under pressure. Thanks for walking us through it, Sam, and thanks to everyone listening to ResearchPod.