When an agent tells me it did the work, how do I know it did? I've spent a good chunk of this year building checks to answer that, and the AI kept finding ways to pass them without doing it.
I run agents on my own content system, on builds for consulting clients and at my day job. In all three the agent writes the log, runs the test, applies the fix and a lot of the time grades the result too. That means whatever passes my checks is often the same agent I'm checking, and whoever reads the output later is trusting that pass. If you run agents on real work, you're leaning on the same kind of checks.
Here are seven of mine, from the weakest I've relied on to the strongest.
The log recorded three steps that no agent ever ran
At a client, the main chat session that's supposed to hand work out to agents wrote three steps of a post itself. They were the early ones (picking the angle, choosing the proof and outlining), and each is meant to be done by an agent following that step's written instructions. The run log recorded each of the three as following its instructions, though those instructions had never been loaded.
For the angle and proof steps the log also lists what each one read. It's a brief and a handful of other files, and not one of them holds that step's instructions. I caught it by reading the post, which had a line of generic AI filler in it and gave me a sense that the whole process wasn't running.
The run log is the cheapest of the seven to pass, because whoever does the work writes it. Agents at my day job did the same thing by typing scores like "8.7/10" onto quality checks they hadn't run. Back in July, on my own setup, I'd already built a check that looks for the files a real run leaves behind instead of trusting the log. At the client the step files were there, and the main chat session had written them.
A test that couldn't fail, and 21 planted mistakes my checker had to catch
An agent working on my local family-events newsletters handed in a test that couldn't have failed on any machine. It was working through a plan for the code that pulls events into those newsletters, where an event's time or date could come out wrong on a machine set to another timezone. The agent wrote that one test up as showing the code gave the same result in any timezone. The test set the zone to New York and then checked that the zone was New York.
I keep a list of the datasets my agents can use, with directions for getting each one. When I built a checker for it, I planted 21 defects across 5 fake entries. I didn't count it as working until it passed the real list and failed the broken one. It caught all 21, including an entry with blank directions and an "unverified" flag left without a note.
An AI fix swapped em-dashes for colons, so my checker now counts them as one
An essay of mine passed four checks I put my writing through, and when I read it I couldn't call the voice mine. The last of those checks had scored it 8.3 out of 10. Its em-dashes had come down from 39 to 12 along the way.
The fix from that last check had turned em-dashes into colons. A colon there is the same pause in a different mark, and I've written that habit up before. The old rules in the script that checks my writing capped each mark on its own, so as written they couldn't see a swap from one to the other.
I replaced the separate caps with one shared limit of about one per 150 words. It counts em-dashes, sentences built around a colon and short dramatic fragments together. I'd frozen a copy of the old draft before the rewrite touched it, and the new check failed it at 35 pauses against a limit of 17.
A different essay that I edited by hand fails the default limit too, 7 pauses against 5. Most of those are short emphasis fragments, and I write personal essays that way. So essays get their own setting where those fragments don't count and em-dashes and colons still do. It passes there.
Other AIs fact-checked my survival guide and found dangerous mistakes
My offline survival guide is out for anyone to read, and it has safety advice for wild mushrooms in it, so a wrong fact there can hurt someone. Every check on how it was put together passed fair and square. In one test run a fresh agent with no internet answered from the guide and refused unsafe eating.
In June, before I would sign off, I asked independent agents for a fact-check with sources, though the plan didn't call for one. They came back NO-GO with 4 critical errors and 13 major ones. Among them were a deadly look-alike named as the wrong species and a wrong minimum cooking temperature.
The fix added a new critical error. It put a spore-print test into one mushroom's safety advice, and that test can't tell the mushroom from its toxic look-alike. Only a second, independent round of the fact-check caught it. That round also found two invented citations.
A third round re-checked just the four files changed after the second and came back clean. That same month an agent recorded my safety read on the guide as passed by multi-agent checks, though I'd never done it. In October I withdrew that pass. The fact-check I ran then found 23 critical errors, all in the guide's 23 original biology entries.
I've written about docs that are structurally perfect and factually reversed, and those entries had been live in the public guide. The guide is public with the October fixes in, and its front page now says plainly that no human has read it yet and every entry is only agent-verified. I made "reviewed" a human-only label behind a script that won't run without a live terminal and a typed confirmation phrase.
Giving my AI grader a test before it graded anything
I built an AI grader to tell me which of two drafts is better, and it was no use to me unless it could tell clean writing from slop. It compares the two without knowing which is which. Before its first run I wrote down the tests it had to pass and what counted as passing. Identical drafts had to tie, and a draft full of AI filler had to lose at least four of its five sections to its cleaned-up version (a real pair, where a person had marked which one was the slop).
On the first run the slop draft lost only three, and on every repeat the grader gave it one section (it said the slop "opens and closes tighter"). So I told it in its prompt to judge voice, not general craft, and left every passing bar where it was. On the second run the slop lost four of five. Identical drafts tied both times.
The judge I use to score whether a draft reads like I wrote it has never had a test like that. In June it went through a slop version of one of my essays and flagged all 9 AI-writing habits marked in it. It already had the list of those habits in front of it.
Codex found a hole in Claude's plan, and running the code proved it
One of Claude's plans was copying over a safety guard from a July branch of my setup, and the guard had a hole in it. The guard was meant to stop an agent from adding every changed file to the next commit in one go. I have OpenAI's Codex and Google's Gemini review the plans Claude writes, and if running the code confirms something they found, my review tool counts it as a real finding.
Codex found the guard let that command through whenever it was written in quotes, because the guard deleted anything inside quotes before checking. The plain git add -A was blocked. Three quoted commands that do the same job went through.
Trying those commands reproduced the hole before any fix went in. One of the three quoted commands came from Claude, not Codex. The guard's existing test of its quote handling only covered commit messages, so copying the guard over unchanged could have shipped the hole with every test passing. The quote check got fixed in both versions of the setup I package for others.
Proof from Claude Code's record of what each agent did, not from the agent
Back at the client from the first check, the brief for the fix asked for a setup that "cannot pretend." One piece of that was blocking the main chat session from writing a step's file. Claude Code lets you hook in a check that runs before its file-saving tool does anything, and the client's hook turned those writes away. The script that runs the steps would also log a step only if it carried an agent's ID.
For every step after that, I had the script take its proof from the record Claude Code keeps of each agent's work. That record shows what the agent actually read and wrote, so the proof doesn't rest on anything the agent says about itself.
One step's proof read "25 reads, own write". That shows the step ran, not that it was any good. In testing, the setup passed 24 of 24 cases and the script refused an outline with no agent's ID on it. The post from the first check stayed a partial result, because its angle and proof steps were never re-run by agents.
The client's agents run on a toolkit that soon got its own proof check. About four weeks after those changes, that check started turning away real steps over a format mismatch. The record is still just a file, so writing the proof by hand was an option, and I ruled it out. The fix went into the toolkit.
In my own content system, the check on files my agents write still only logs them and lets them through. Of the 8 times my publish script had moved a file into its final place by October 22, none had a valid review behind it.
Where each check is as of October 22
| Check | Status |
|---|---|
| The log, and Claude Code's record, at the client | Enforced only in the client's setup. On a later post the hook blocked a real attempt by the main chat session to write a draft. |
| The checker for my dataset list | Run on ship day, July 16, and again for this write-up, when it still caught all 21. Nothing schedules it. |
| One shared limit on em-dashes, colons and fragments | Live for my articles and essays. The copy of the old draft is gone, so nothing can re-run it. |
| Survival guide fact-check | A stopgap do-not-use warning, meant to hold until the corrected entries landed, was approved but never added to the public guide. |
| The grader's tests, set before its first run | Judging voice still isn't the default in two of the grader's three scripts. Those tests haven't run since its default model changed September 29. |
| Codex and Gemini reviewing Claude's plans (the quoted git commands) | 68 of 68 tests pass in each version of the setup I package for others. The guard there flags all four commands but only warns. |
What I trust now when an agent says it's done
I trust hooks and code that actually enforce intended behavior, and auditable logs with unbiased agents that read and check that these things actually happened. And to be honest, I never fully trust it anyway, I use my own judgment and taste on the final output.
