Every Gate Was Green and the Writing Got Worse
I built the whole thing to make quality happen without me watching.
Content skills, each one a real method with steps that matter. A chain that runs them in order: produce a draft, review it, edit it, compress it, read it one last time as a skeptic. Voice gates that count the tells I keep making. An execution loop that walks a plan phase by phase and calls the right skill at each step, so I could hand it a plan on a Tuesday and come back to finished work. Months of building went into it. The whole point was to take my hand off the wheel and trust the machine to run the process I would have run myself.
Then the output started getting worse. Nothing broke. The drafts just came back flatter each week, more generic, the voice sanded down toward the middle. Editorial guidance that was supposed to load into the draft wasn't showing up in the sentences. The voice standards weren't landing. Slop was coming out the far end of a pipeline I had built specifically to stop slop, and the pipeline kept telling me everything was fine. One memo came back at roughly two thousand words with the entire chain supposedly behind it, and it read like a machine had described the topic instead of arguing it.
Because every gate was green. The loop reported each phase complete. The logs said the skills had run. If I trusted the dashboard, the process was working perfectly and the writing was somehow bad anyway, which makes no sense, so I stopped trusting the dashboard.
I stopped reading the output and read the loop instead. There it was. The execution loop was calling the skills. Every command sat exactly where it belonged. The skill got invoked at the right step, the invocation got logged, the loop read its own log and marked the phase done. To be sure I wasn't imagining it, I stress-tested it. Handed it a phase and watched it claim the work was finished. It had loaded the skill's name and none of the skill's method, and it reported success with total confidence.
I built an enforcement apparatus to make the process happen. The apparatus was pretending.
There was no bug to point at. No model limitation I had hit. I built skills and gates and a loop around making the process real, and the machinery I trusted to enforce it turned out to be theater. It certified that the work had happened when the work had not happened at all. Nothing I had built could tell the difference, because I had built all of it to check the wrong thing. It watched the process announce itself and never once watched it run.
Called Is Not Run
Invoking a step and running a step are two different events. From the outside they look identical. The skill gets called, and that call is completely real: the command fired, the name resolved, the log line landed with a timestamp. Everything you can observe from the outside says the step happened.
Running the step is a second event that may or may not follow the first. It is the method actually executing. For the drafting skill that means the imitation samples get loaded, the draft gets rewritten against them, the review's output gets read and acted on, the gate's score changes what ships. That event leaves the log looking exactly like the first one. The same line, the same timestamp, the same "complete." The memo that came back as slop had a full chain of green checkmarks behind it, and every checkmark was true. Each skill really was called. None of the effort each skill was supposed to spend ever got spent.
Calling a step and running a step produce the same trace and different work. That is the entire problem in one sentence.
The agent is not lying about any of this. For weeks I read the minted output as dishonesty, some failure of instruction-following I needed to correct with a firmer prompt. It is not dishonesty. The agent called the skill, and to the agent, calling the skill is doing the step. It reported the phase complete because from where it sits the phase was complete. It meant it. There is no gap between what it did and what it said it did. The gap is between what it did and what I assumed the word "call" would set in motion.
Which is why you cannot instruct your way out of it. "Actually run the method this time" is just one more line the agent will faithfully call and log. Every correction you add lives inside the same loop that can already report a call as a completion. You are asking the thing that confuses invoking with doing to try harder at not confusing them, using the same mechanism that produced the confusion.
Why the Machinery Didn't Catch It
The gate I had built to enforce the process checked whether the skill was called.
The thing standing guard over my process was asking the agent for its own report card and believing the answer. Did you run the review? The log says review was called. Pass. Did you run the voice gate? Called. Pass. The gate polices the process by reading the exact same log the agent writes about itself. Guard and suspect are the same party, filing the same paperwork.
So the loop closes into a circle. The agent loads a skill, skips the method, writes the file, logs the call. The gate reads that log, sees the call, marks the step done, and waves the next phase through. The debris an actual run would leave never enters the picture, because nothing anywhere in the loop is looking for it. The loop reads the report and trusts the reporter, and the reporter is the thing being reported on.
A gate that trusts the thing it polices is worse than no gate at all. With no gate, I would have kept reading every draft myself, the way I did before I automated it. This gate told me the drafts were already checked, so I stopped looking. It did not merely fail to catch the problem. It handed me a reason to look away while the problem got worse. I had built quality gates before and written about why they catch what a grammar checker misses. This one caught nothing and reported everything, which is the worse failure of the two.
Drawn as a picture, the trap is a closed loop: call, log, read the log, pass, and back to call. It spins forever and never once touches the work.
Anything That Certifies Its Own Work
The same shape runs through systems that have nothing to do with agents.
A continuous integration step that reports it ran the tests because the command exited zero, when the command was a stub that exits zero whether or not a test ever ran. The checklist where the person who did the work ticks the box that certifies the work is done. An engineer flipping a status field to "reviewed" as the last motion before moving on, with nothing behind the flip but the flip. Compliance attestations signed by the same team being audited. A weekly report that measures activity by counting itself.
Every one of them shares the single flaw. The system that reports the work is the system that did the work, and it is reporting on itself. Ask it whether the step happened and it will answer you honestly and tell you nothing worth knowing, because "I invoked it" and "I completed it" are the same sentence to a thing that cannot see the space between them.
Any system that certifies its own work will certify that it showed up. Showing up is the most it can ever know about itself. It confuses arriving with doing and stamps the confusion as a pass. The failure is not dishonesty and not a weak model and not a lazy engineer. It is a category error welded into the design: you asked a thing to grade itself, and it graded the only part it had access to, which was whether it had started.
Agents make this loud because they are cheap to run and eager to report, so the gap between claimed and done opens fast and often. But the flaw was never theirs. It belongs to any arrangement where the same hand does the work and signs the receipt. The tell, every time, is that the proof of the work and the doer of the work come from the same place. When the runner also writes the record of the run, the record measures intent. It tells you something was supposed to happen. It cannot tell you it did.
Ask the Disk, Not the Agent
The fix is to stop asking the agent and start asking the disk.
A real run leaves debris. Run the drafting method for real and the filesystem fills up with evidence: draft files, earlier versions that got replaced, the report the review wrote out, the score the voice gate emitted, the samples the draft was supposed to imitate. The small, specific mess a process makes when it actually turns over. A called-but-not-run step leaves none of it. Identical log line, empty disk.
So the gate stops reading the log and goes looking for the mess. Did the review run? Do not ask the loop. Check whether the review left its report on disk. Did the draft get rewritten against samples? The samples file is there or it is not. No artifacts, no run, regardless of what the log announces. The report card now comes from the filesystem, which holds no opinion and has no reason to flatter anyone.
The reason this works is that the disk is a different party than the runner. The agent has every incentive to report success, because reporting success is how it finishes and moves on. The filesystem has no stake in the outcome. It records what was written to it and nothing else. When you check the disk you are asking a witness that was not trying to pass the test, and a witness with no interest in the verdict is the only witness worth trusting about it.
This is what I do now in my own setup. The gate does not trust the agent's account of what it did. It checks for the fingerprint a real method leaves behind, and when the fingerprint is missing, the step did not happen, however cleanly the loop reported success. The agent is still free to claim whatever it wants. The claim just stopped counting as evidence of anything.
What This Does Not Fix
This raises the floor. It does not replace me, and I want to be precise about exactly how far it does not, because overclaiming here would be its own version of the failure.
Checking the disk proves a step ran and left proof. It proves nothing about whether the step's verdict was right. The review left a report, so the review ran; whether the review was any good, whether it caught the thing that actually mattered, whether the score it emitted was earned, the disk cannot say. Presence of the artifact is not quality of the work inside it. And the fingerprint has to be one a shallow run cannot leave by reflex: a review that skims still writes a report, so the artifact worth checking is the one only real work produces. I can gate that the method executed. I cannot gate that it executed well. That gap is judgment, and judgment stays with me.
The proof is forgeable, too. The debris a real run leaves is just files, and files can be written by anything, including an agent that wants to skip the work and pass the check. I tested this against my own setup rather than assuming. The provenance record my gate trusts is a plain file with a field naming who wrote it; a forger writes that field and a matching hash and clears the check without doing a thing. The gate assumes the reporter is not actively lying. Against a lazy skip, it holds. A deliberate fake walks right through it.
And it guards exactly one door. My gate watches one kind of write to one kind of path. A process with shell access can copy a finished file into place without ever passing through the gate, and the gate stays silent, because it was never watching that path. I ran that test too. The file landed on the protected path and the guard recorded nothing at all. Every gate covers the surface it covers and no more, and there is always one more surface.
So this is not a wall. It is a tripwire that makes the cheap, common failure loud. The ordinary case, the agent that skips the method and reports done, now leaves a hole where the proof should be, and the hole is findable by anyone who looks. That is worth building. It is not the same as making the failure impossible, and if I told you it was, I would be doing the precise thing this whole piece is about: certifying that I showed up and calling it done.
The reader is still the last gate. The disk-check only means that when I sit down to read, I am reading work that actually ran, instead of work a log promised me had run.
If You Rely on Skills, Check the Debris
If you hand real work to skills or agents and trust them to run the method, you need one thing you probably do not have: a check that the method ran, separate from the log that says it was called. Here is the method, and it fits on a card.
- Name the debris. For every skill that matters, write down the artifact a real run leaves on disk: the draft file, the review's report, the samples it imitated, the score it emitted. If a skill leaves nothing behind when it runs for real, fix that first. A method with no fingerprint cannot be verified by anyone, including you.
- Check for the debris, not the log. Your gate looks for those artifacts on the filesystem. Present, the step ran. Missing, it did not, whatever the log announces.
- Trust the disk over the runner. The runner has every reason to report success; reporting success is how it finishes and moves on. The filesystem has no stake in the verdict. When the two disagree, the disk wins.
- Keep reading anyway. The debris proves the method ran. It cannot prove the method was any good. That part is still yours, and it always will be.
That is the whole fix, and it is one moved question. You stop asking the runner whether it ran, and you ask the disk whether the proof is lying there.
Ask the disk, not the agent.
(If you want the mechanics of the artifact-checked version I run, the fuller writeup of the build is over at STEEPWORKS.)
