This paper reports a single, fully instrumented case study of a large-scale architectural refactoring by an AI coding agent under a specification-first protocol, with no human review of the generated code and no pre-existing oracle to validate the target behaviour. The task, dismantling a central invariant across a large interdependent codebase, was assessed by the author as effectively infeasible through incremental refactoring, the kind of change that conventionally calls for a rewrite instead. Under the protocol described here, the agent completed it successfully. The system is a 717,725-line production TypeScript application across 3,648 files. The task required dismantling a core lifetime invariant: the guarantee that a UI panel remains open for the duration of an AI request. The target behaviour was that a streaming generation survives the closing of its panel and can be reattached, on reopening, to the same live stream with no loss or duplication. The protocol: formal specification by the agent, 14 refinement cycles auditing that specification against the source code, atomic implementation, a compile/test feedback loop, then 17 verification cycles auditing the code against the frozen specification. Across 31 audit passes, 201 defects were corrected before any human executed the program. The convergence criterion was empirical: two consecutive verification passes returning zero findings. The change touched 189 files (31 new); with the extraction phase, the two commits total 288 files, 34,770 insertions, 16,422 deletions. Across the first and roughly thirty later sessions, the software behaved as specified, no bug observed. Elapsed: three days; cost: USD 2,430. The full specification and raw session logs, 1,500+ pages in French, are published as evidence, allowing inspection of the process and submission to a language model for consistency checking.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at how an AI coding agent performed a major software update — without any human reviewing its work.
Sam: That's right. This paper examines what the researchers call "specification-first refinement." The core idea is to force the AI to write a detailed plan before it touches a single line of code — rather than writing code and hoping a human catches the mistakes afterward.
Alex: That's a real shift. Normally, a developer writes the code, and then someone else reviews it. Here, the plan comes first?
Sam: Exactly. And the reason that matters is scale. In large software projects, a single change can ripple through hundreds of files. No human reviewer can realistically trace every connection. So the researchers asked: what if, instead of relying on a person to catch errors at the end, you forced the AI to commit to its intent at the beginning?
Alex: So what was the specific task they gave the AI?
Sam: They asked it to remove what's called a "lifetime invariant." In plain terms, the software had a rule that said: a certain panel — think of it like a window on your screen — had to stay open for a background process to keep running. The researchers wanted to break that dependency, so the data could survive even if the user closed the window. The catch was that this behavior had never existed before. There was no existing test to check whether the AI got it right.
Alex: If there's no test, how does the AI know it hasn't broken anything?
Sam: That's exactly the problem. Normally, you'd have what researchers call a "verification oracle" — essentially a master answer key. A test suite that runs the code and tells you: yes, this works, or no, it doesn't. But when you're building something that has never existed, you don't have that key. You have to construct your own safety net.
Alex: So how did they do that?
Sam: They built what they call a "governance substrate." Think of it as a living rulebook — one that grows every time the system runs into a problem. It includes automated tools that scan code for logical contradictions, and documentation that acts as guardrails, keeping the agent within safe boundaries. Crucially, whenever the agent hit a snag during development, the researchers didn't just fix the immediate problem. They turned that failure into a permanent new rule in the rulebook, so the same mistake couldn't happen again.
Alex: So it's like a checklist that gets longer and more specific every time something goes wrong?
Sam: That's a good way to put it. And the agent operated under a very large version of that checklist — a set of instructions roughly the length of a short novel. Every constraint in there existed because someone, at some point, had seen what happens when that constraint is missing.
Alex: Does that mean the agent is constantly checking its own work against this rulebook?
Sam: It does. The process works in two phases. First, before writing any code, the agent drafts a detailed specification — a blueprint of exactly what it intends to do. Then it enters what the researchers call a "refinement cycle," where it compares that blueprint against the actual existing codebase, looking for conflicts. In this study, the agent caught and corrected over two hundred potential problems during that phase alone, before a single line of new code was written.
Alex: And then once it starts writing?
Sam: Then it enters a "verification loop." It writes the code, then audits that code against the frozen specification it committed to at the start. The key word is frozen — the specification doesn't change. The agent keeps auditing until it passes two consecutive checks with zero findings. Only then does it stop.
Alex: So it's proofreading its own work — but against a fixed target it can't move?
Sam: Exactly. And that distinction matters a lot. Research has shown that if you ask an AI to simply revise its own output — without any external reference point — it tends to drift. It might "fix" something that wasn't broken, or convince itself the work is done when it isn't. By locking the specification in place first, the agent has an anchor it can't rewrite. It's not asking itself "does this feel right?" It's asking "does this match the plan I already committed to?"
Alex: That's a meaningful difference. It's not about having a second AI watching the first one — it's about having a fixed goal the agent is held to.
Sam: Right. And the result, in this case, was that the agent refactored close to 190 files across a codebase of over 700,000 lines of code — in three days. The final build worked as specified, with no bugs observed, and no human reviewed the code.
Alex: That's a significant result. But I want to push on the limitations here. How do we know this wasn't a one-off?
Sam: That's the right question to ask, and the paper is honest about it. This is a single case study. There's no control group — no version of the same task done the traditional way to compare against. The results are also self-reported by the researchers who built the system. So while the outcome is notable, it's one data point. We can't draw broad conclusions from a single experiment.
Alex: So the honest summary is: this approach worked here, in this specific context, and it's worth taking seriously — but it hasn't been proven to generalize?
Sam: That's a fair reading. What the paper does establish is a proof of concept. It shows that with the right structure — a frozen specification, a living rulebook, and a convergence-based auditing loop — an AI agent can handle a complex, large-scale task without continuous human supervision. Whether that holds across different codebases, different AI models, or different types of changes is still an open question.
Alex: It does suggest a different way of thinking about the problem, though. Instead of asking "how do we get more humans to review AI output," you ask "how do we structure the AI's process so the output is trustworthy before it reaches a human?"
Sam: That's the shift the paper is pointing toward. The burden moves from after the fact — someone checking whether the code looks right — to before the fact — the agent proving its work matches a plan it was held to from the start. Whether that scales, and under what conditions it breaks down, is where the research needs to go next.
Alex: A clear-eyed result, and an honest set of open questions. Thanks for walking us through it.
Sam: Thanks for listening to ResearchPod.