The AI Codebase Maturity Model: From Assisted Coding to Self-Sustaining Systems

AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.

Sam: Anderson draws from the Capability Maturity Model, or CMM, created in 1993 by the Software Engineering Institute. It laid out steps for improving software processes, where each stage builds on the last—like laying bricks only after the ground is solid. Its update, CMMI, spread that idea wider, but the key was predictable progress through checks at every level. He also adapts DORA metrics for DevOps teams—like how often they deploy code, how long changes take, how often updates fail, and how quickly they fix problems. These link directly to better performance through real outcomes.

Alex: Huh, so it's less about the team and more about what the code enforces. How does his model stand out from newer AI ones?

Sam: Most AI maturity models chase autonomy—how much humans step back. They overlook that without tests, metrics, and auto-fixes encoded right in the repo, more freedom just breeds errors. His ACMM flips it: maturity means the codebase holds the smarts through 33 feedback loops, like instructions and self-tuning configs, turning chaos into 24/7 fixes under 30 minutes—proven in his four-month build.

Alex: That distinction makes sense—the infrastructure teaches the AI, not the model alone. So the real intelligence is baked into the tests and loops. Walk me through how those levels actually play out in practice—what changes from one to the next?

Sam: At the first level, you prompt the AI for each bit of code and review everything yourself, like asking a helper for suggestions one at a time without any shared notes. Nothing carries over between chats, so you repeat instructions every time. The next step writes those repeated tips into files the AI reads at the start of every session—like a rulebook everyone follows, making outputs steady without you saying the same thing over and over.

Alex: Okay, so instructions cut down on repetition. But how do you know if it's actually getting better, not just consistent?

Sam: You add measurements, like tracking how often AI suggestions get accepted or if tests pass. Nightly checks run automatically, spotting issues before they grow—think of it as a report card that shows patterns in what works. Testing stands out here: piles of test cases, high coverage so most code gets checked, and reliable runs without random failures. A shaky test wastes time for people, but for AI running alone, it breaks trust entirely.

Alex: Huh—that's why testing was the big investment. So once you measure, what happens next?

Sam: The system starts acting on those numbers automatically—like blocking weak AI tasks if acceptance dips below a set point, or fixing bugs overnight. Humans watch over, but don't push every button. At the top level, the whole setup runs 24/7: community issues turn into fixes fast, docs update themselves, and the code refines its own rules from past successes.

Alex: Wait, so the smarts aren't in the AI tool itself... It's all in those files, tests, and loops around it. How did that play out week by week in this actual project?

Sam: In the first couple weeks, it was all manual prompting per session, leading to repeated mistakes since nothing carried over. By weeks three and four, he created files like CLAUDE.md—a document loaded at every AI chat start, listing rules from past errors, such as "don't change unrelated code" or "use named constants instead of random numbers." This acted like a shared notebook, cutting repetition and stabilizing outputs without constant human reminders.

Alex: That notebook idea keeps things consistent across sessions. But you mentioned measurements next—did that reveal specific patterns?

Sam: Weeks five through eight focused on tests: 32 automated nightly runs checking everything from performance to accessibility, hitting 91% coverage so almost all code got verified. A file called auto-qa-tuning.json tracked how often AI pull requests got merged versus rejected by category, spotting weak spots like low acceptance for certain tasks. Flaky tests—ones that pass or fail randomly—got hunted down weekly, because unreliable checks block good fixes or let bad ones through in an AI-only flow.

Alex: Huh, so deterministic tests are non-negotiable for autonomy. Then the loops closed—how?

Sam: In weeks nine to twelve, automation kicked in: the system adjusted task priorities automatically—if accessibility changes merged 62% of the time, it favored them more, while blocking poor performers like operator updates at 8%. Loops scanned repos every 15 minutes for issues, monitored builds constantly, and handled user bugs end-to-end, from report to fix preview, all under human oversight but without daily intervention.

Alex: Wait—user bugs to fixes without a person typing? Give me an example.

Sam: At level five, a user reported a cluster marked healthy despite pod issues. The codebase—tests, docs, logic—distinguished it as a misunderstanding of Kubernetes design, where cluster health checks nodes separately from workloads, and auto-explained that before any human saw it. Humans shifted to high-level steering, like governance, not line-by-line reviews.

Alex: So the system judged bugs versus confusion on its own... A clear improvement in reliability. What did that look like by the final weeks—once it hit full self-sustaining mode?

Sam: In weeks 13 to 16, issues reported at 2 AM got triaged, fixed, tested, and ready for review by morning. The system analyzed its own past successes weekly, tweaking rules like keeping changes small for better acceptance. Automated workflows even created tutorials from new features—grabbing screenshots and generating narrated slides—and synced docs with every merged update. No single tool made it happen; the wired-together pieces ran nonstop. A full count shows 33 distinct loops, from session-start instructions to hourly error checks and 15-minute issue scans across repos.

Alex: So it handled the whole cycle without you overnight. Huh, the loops caught real issues and fakes alike. Any numbers on how reliable it got overall?

Sam: Over 82 days, it merged over 80% of pull requests and closed nearly all issues, with bugs turning into fixes in about 30 minutes around the clock. The key insight: smarts live in those instructions, tests, and loops—not the AI model, which could swap out easily.

Alex: Right—the system's the brain... Makes sense why infrastructure matters most. So infrastructure like tests and loops keeps technical debt from piling up—messy code that slows everything down over time, like a cluttered room where you can't find tools. How does that play out day-to-day once it's running smoothly?

Sam: Clean code pays off because AI works better on structured projects—refactoring, which means rewriting parts without changing their job, just to make them neater, boosts every future change. Telemetry acts like a constant watchful eye: it tracks errors hourly, user drop-offs, and feedback right away, spotting breaks before reports come in and kicking off fixes automatically. Without that sensing layer, the system runs but misses problems, staying blind to real use. For open source, anyone files a simple issue describing a bug or wish, and the setup handles it end-to-end: triage, code, tests, docs, often in 30 minutes to an hour, day or night. Users become the roadmap without touching code; humans just guide big choices.

Alex: Huh, so monitoring turns it into something that feels alive. That shifts open source from volunteer grind to shared direction. But the paper flags some limits—what are they?

Sam: It's one case: a solo dev on a dashboard app, not safety-critical code or complex algorithms, so it may not fit teams or high-stakes areas. Survivorship bias means we see success, not the failures that dropped out; no broad testing across projects yet.

Alex: Fair point—keeps it grounded as a single example. Still, the advice stands out: start simple with rule files and tests, build measurements, let loops handle the rest.

Sam: Exactly—the paper stresses testing as the core trust builder, with infrastructure compounding gains level by level. Most using AI code sit at early steps; knowing the path forward matters more than rushing. Infrastructure holds the judgment; AI just executes.

Alex: The code remembers, tests catch slips, loops stay honest... That's a solid takeaway from this look at AI codebase maturity. Thanks for breaking it down, Sam. Thanks for listening to ResearchPod.

Listen to the full podcast episode →