Joël Abenhaïm
6 min
Abstract
This paper reports a single, fully instrumented case study of a large-scale architectural refactoring by an AI coding agent under a specification-first protocol, with no human review of the generated code and no pre-existing oracle to validate the target behaviour. The task, dismantling a central invariant across a large interdependent codebase, was assessed by the author as effectively infeasible through incremental refactoring, the kind of change that conventionally calls for a rewrite instead. Under the protocol described here, the agent completed it successfully. The system is a 717,725-line production TypeScript application across 3,648 files. The task required dismantling a core lifetime invariant: the guarantee that a UI panel remains open for the duration of an AI request. The target behaviour was that a streaming generation survives the closing of its panel and can be reattached, on reopening, to the same live stream with no loss or duplication. The protocol: formal specification by the agent, 14 refinement cycles auditing that specification against the source code, atomic implementation, a compile/test feedback loop, then 17 verification cycles auditing the code against the frozen specification. Across 31 audit passes, 201 defects were corrected before any human executed the program. The convergence criterion was empirical: two consecutive verification passes returning zero findings. The change touched 189 files (31 new); with the extraction phase, the two commits total 288 files, 34,770 insertions, 16,422 deletions. Across the first and roughly thirty later sessions, the software behaved as specified, no bug observed. Elapsed: three days; cost: USD 2,430. The full specification and raw session logs, 1,500+ pages in French, are published as evidence, allowing inspection of the process and submission to a language model for consistency checking.
Sam: That's a good way to put it. And the agent operated under a very large version of that checklist — a set of instructions roughly the length of a short novel. Every constraint in there existed because someone, at some point, had seen what happens when that constraint is missing.
Alex: Does that mean the agent is constantly checking its own work against this rulebook?
Sam: It does. The process works in two phases. First, before writing any code, the agent drafts a detailed specification — a blueprint of exactly what it intends to do. Then it enters what the researchers call a "refinement cycle," where it compares that blueprint against the actual existing codebase, looking for conflicts. In this study, the agent caught and corrected over two hundred potential problems during that phase alone, before a single line of new code was written.
Alex: And then once it starts writing?
Sam: Then it enters a "verification loop." It writes the code, then audits that code against the frozen specification it committed to at the start. The key word is frozen — the specification doesn't change. The agent keeps auditing until it passes two consecutive checks with zero findings. Only then does it stop.
Alex: So it's proofreading its own work — but against a fixed target it can't move?
Sam: Exactly. And that distinction matters a lot. Research has shown that if you ask an AI to simply revise its own output — without any external reference point — it tends to drift. It might "fix" something that wasn't broken, or convince itself the work is done when it isn't. By locking the specification in place first, the agent has an anchor it can't rewrite. It's not asking itself "does this feel right?" It's asking "does this match the plan I already committed to?"
Alex: That's a meaningful difference. It's not about having a second AI watching the first one — it's about having a fixed goal the agent is held to.
Sam: Right. And the result, in this case, was that the agent refactored close to 190 files across a codebase of over 700,000 lines of code — in three days. The final build worked as specified, with no bugs observed, and no human reviewed the code.
Alex: That's a significant result. But I want to push on the limitations here. How do we know this wasn't a one-off?
Sam: That's the right question to ask, and the paper is honest about it. This is a single case study. There's no control group — no version of the same task done the traditional way to compare against. The results are also self-reported by the researchers who built the system. So while the outcome is notable, it's one data point. We can't draw broad conclusions from a single experiment.
Alex: So the honest summary is: this approach worked here, in this specific context, and it's worth taking seriously — but it hasn't been proven to generalize?
Sam: That's a fair reading. What the paper does establish is a proof of concept. It shows that with the right structure — a frozen specification, a living rulebook, and a convergence-based auditing loop — an AI agent can handle a complex, large-scale task without continuous human supervision. Whether that holds across different codebases, different AI models, or different types of changes is still an open question.
Alex: It does suggest a different way of thinking about the problem, though. Instead of asking "how do we get more humans to review AI output," you ask "how do we structure the AI's process so the output is trustworthy before it reaches a human?"
Sam: That's the shift the paper is pointing toward. The burden moves from after the fact — someone checking whether the code looks right — to before the fact — the agent proving its work matches a plan it was held to from the start. Whether that scales, and under what conditions it breaks down, is where the research needs to go next.
Alex: A clear-eyed result, and an honest set of open questions. Thanks for walking us through it.
Sam: Thanks for listening to ResearchPod.