I spent six months building release engineering for my AI agents. A January paper just named it.

Last Friday the C-Suite framework’s weekly system eval logged 3 fails, down from 13 the Friday before and 18 the Friday before that. The pass count on the same 50-check denominator climbed from 22 to 37 to 47 across the same three weeks. The agents are not getting smarter on their own. The discipline behind the line is catching the failures the prior week surfaced and refusing to let them ship.

I have been building AI agents on this discipline for six months without a name for it. In January a paper landed on arXiv called AgentDevel, and it named the thing: regression-aware release engineering for self-evolving LLM agents. The vocabulary it ships with (single canonical version line, implementation-blind symptom-level quality signals, flip-centered gating that prioritizes regression prevention) is the academic restatement of what I had been arriving at on instinct.

This is what the practice has actually looked like.

What had been in my head for six months

The C-Suite framework I run for my own consulting practice is a 15-agent system: a Chief of Staff agent orchestrating the rest, plus specialist roles (Mechanic does code review, Client Success does UX review, Librarian does external research, on through the roster). Every Friday I run a Kaizen pass: I pull a sample of last week’s outputs, score them against a written rubric, and ask what the system needs to learn to do better next week.

For most of the six months I was scoring this by feeling. The Friday pass felt productive, the next week’s outputs seemed sharper than the week before, and the loop felt like it was closing. Felt was the operative word.

The thing that changed three Fridays ago is that I added a fixture suite. Five recent deliverables get sampled each Friday and run against a 50-check rubric scored by Sonnet 4.6 with operator spot-check calibration. Each check is yes or no. Pass count gets logged to a JSONL file, and the line on disk answers the only question I actually care about: is the system getting smarter or is it decaying?

The trajectory reads 22 pass, then 37, then 47, against the same denominator. The fail count fell from 18 to 13 to 3. That is not a vibe. That is the discipline catching the failure modes the prior week surfaced and refusing to let them ship.

The paper that named it

In January a group of researchers published AgentDevel: Reframing Self-Evolving LLM Agents as Release Engineering (arXiv:2601.04620). The argument is structural. Most attempts at agent self-improvement embed the improvement loop inside the agent itself, which produces unauditable behavior change. The alternative the paper proposes is to externalize the loop into a regression-aware release pipeline: execute the current version, generate implementation-blind symptom-level quality signals from execution traces, synthesize a single release candidate via executable diagnosis, deploy via flip-centered gating that prioritizes regression prevention. Every change to the agent lives on a single canonical version line with reproducible, auditable artifacts.

If you read that paragraph and squint, it is a description of Friday Kaizen on the C-Suite framework. The “implementation-blind symptom-level quality signals” are the binary fixtures (did the output pass the rubric, did the tool sequence complete, did the response cite the right source, the things a user would notice). The “single canonical version line” is the pinned prompt, model, and tool set that each weekly pass runs against. The “flip-centered gating that prioritizes regression prevention” is the rule that a previously-passing fixture flipping to fail blocks the release.

The paper did not change what I was doing. It gave me a name for it that I can use in a panel without having to walk the audience through a 20-minute explanation.

The four principles, mapped

The Google SRE book, chapter 8, names four principles for software release engineering. AgentDevel inherits them. The mapping to AI agents is direct.

Self-service deploys means any prompt edit, tool change, or model swap goes through the same eval suite. The eval is the gate, not a human reviewer who happens to be available.

High velocity means weekly Kaizen as the floor cadence. The trajectory line is what makes regressions visible the moment they arrive, not when someone notices the agent feels worse.

Hermetic buildsmeans every artifact pins prompt version, model ID, tool definitions, and eval suite. If a regression shows up, the prior-known-good version is reproducible. Without pinning, “the agent feels worse” has no diagnostic path.

Policy enforcement means regressions block the merge. Manual override is permitted, but leaves an audit entry. The override count itself is a metric.

Most operators I talk to are running zero of the four with the discipline visible. A senior operator runs all four with the work on disk.

The failure that taught me what the discipline is for

The week the trajectory line went from 37 to 47, the framework almost shipped a doctrine violation.

I was eight bugs deep into a staging-smoke session on a separate consultants-suite build. Three of the bugs were serverless-incompatibility shapes (setTimeout in a batcher, Pino worker-transport, an env-scope assumption). My active-build doctrine has a rule for this: three same-area bugs in 14 days triggers a Rule 10 design review, dispatch to the Mechanic agent with the question “what architectural shape makes this class of bug impossible?”

The Rule 10 trigger got named correctly. The wrong response channel got picked. Instead of Mechanic, the in-session Chief of Staff invoked the codex-prep skill to package a codebase grep sweep for the Codex CLI. Codex was cheaper. The day had run long. The decision felt fine at the cost-axis and was invisible at the discipline-axis. That is the shape of a doctrine miss.

I caught it on the next turn: “don’t we have systems built in for this stuff tho? As per the debugging protocols?”

That got filed as bug-pattern fixture 2026-06-25-codex-prep-used-when-mech-dispatch-was-doctrine-prescribed.md. Symptom on top, root cause underneath, fix and prevention named. The fixture also became one of the 50 binary checks the next Friday Kaizen runs against. If the Chief of Staff makes the same call again, the rubric will catch it before anything else ships.

That is the entire reason the fail count dropped 18 to 13 to 3. The agents did not get smarter on their own. What changed is that bad days now end with a fixture, and fixtures cannot silently return.

Porting it to a system that does not have it yet

The next move I owe myself is installing the discipline on a production system that has no weekly eval cadence yet. That system is opportunities-engine, the agent I built for consultants who serve multiple nonprofit clients. It scores grants and RFPs against each consultant’s tenant fit.

The plan is the loop in miniature: enumerate every Anthropic call site in the codebase, pick the highest-impact operation (the grant scorer, because a bad score wastes a consultant’s afternoon), design 5 binary fixtures from real production inputs in the last 30 days, evenly split between passing and failing examples plus one edge case. Extend the eval runner from the C-Suite framework or stand up a minimal equivalent. Run the baseline, identify the highest-risk regression vector, commit the artifact, set the cadence to run on Friday alongside C-Suite Kaizen.

The value of the discipline being portable is the thing I want operators to see. Once the loop is in your hands, you can install it on any AI system. The four principles travel with it. What you rebuild for each new context is the fixture suite, with its rubric and its pinned model. The line on disk that answers “is this getting smarter” stays the same shape.

This is what the AI Operations practice looks like at a senior register. The agent is a deliverable, but the discipline behind the agent is what clients are actually paying for. “I installed the discipline that catches the agent decaying before your users do” is the sentence I want carrying weight in the room.

What this is not

A few things this is not.

• It is not a claim that the C-Suite framework is the best agent system on the market. I built it for myself. It happens to be the system I use to run my practice, and the fixture suite is the receipt that it is getting better. I have not benchmarked it against anyone else’s.
• It is not a claim that AgentDevel is the only paper worth reading on this. Google SRE book chapter 8 is the canonical foundation. Hamel Husain’s evals FAQ is the operational methodology. Eugene Yan’s seven-pattern reference is the surface map. AgentDevel is one entry in a reading list.
• It is not a claim that you need a Python service surface or a trace store to start. The C-Suite framework runs on Node, bash, and Markdown. The fixture suite is JSONL on disk. A weekly score, a written rubric, and a calibration spot-check is enough to start a trajectory line.
• It is not a claim that the discipline removes the need for a human in the loop. The opposite. The discipline is what tells the human where to look. The whole reason the line moves is that the operator catches what the system ships and feeds the failure modes back as fixtures.

What I keep coming back to

Most AI consultants will tell you their agents are good. The thing I want to be able to show clients is the line on disk that catches the agents getting worse before the client does. That is the senior move.

When a client asks the question I want every kariops engagement to be answering ( “how do I know your system will still be working in three months”), the answer I want to give is not a story. It is a URL. The trajectory file lives in the repo. Anyone with access can read the line week over week.

Six months of building the practice gave me the loop. The January paper gave me the language. The move I want to keep making is installing the loop on every production system in the stack that does not yet run a trajectory line of its own.