In September I was rebuilding the selector that ranked articles for my newsletter, and one of its planned inputs was a novelty score, meant to push up what was new for readers. Before it ranked a single article, I checked it against the ones I'd already picked myself.
Other systems I built hand me numbers that decide things too. My repo has a health score, I built two scores for a client, and there were figures on how much my session digests recorded. One of the client's two, an account score, was meant to tell its team which accounts to work first, and the digests had to earn their keep or go.
A scorer with a bug in it can still hand back a number, and that number looks every bit as exact as a right one, decimals and all. I tested every one of these numbers, and not one of them was measuring what I needed it to.
Ranking newsletter articles by what was new did no better than random
I used to pick the newsletter's articles by hand every day (I've since stopped). I pulled 158 of those picks, mid-March through late July, had each score rank everything that came in the same week as each pick, and then looked at how high my picks sat. A random order would leave them halfway down on average, at 0.50.
Novelty put my picks at 0.451, so for choosing my articles it was a coin flip. The old relevance score got 0.583, and similarity to my earlier picks came out on top at 0.676.
The test leaned toward relevance, since I'd made those picks from the daily list I was shown, which likely favored high-relevance articles. I also counted how many of the 158 made their own week's top twenty. Random would catch about 20. I only tried one combination, relevance plus similarity, and it caught 60.
I got asked in October what I thought when I saw that, and this is what I said.
That we hadn't built evals or calibration on what "good" looks like, and that we hadnt gotten to a place of replacing taste yet in terms of how articles relate over time to broad themes, and unrelated but relevant news (e.g. global news)
I'd also planned to have a model write the answer key for later tests (the other checks I've run on an AI grader are in an earlier piece), so I first checked its calls against 60 I'd made myself. It matched my call on 61.7% of them. The misses went both ways, articles I'd dropped that it wanted to develop and picks of mine it would have held back. When I gave it 14 of my picks as examples it did worse, 53.3%, and I turned the model-written answer key down.
The selector I rebuilt in September left novelty out. Its top twenty caught my picks 1.13 times as often as the old selector's did, on June and July weeks it was never tuned on. I'd set the bar at 1.20 before the run, so I wrote it up as a miss.
My repo's health score came in at 35 out of 100, and the broken part was the scorer
I audited the repo my agents work in back in July. I wanted its health score to tell me whether an agent could find its way around my docs, and the 82 the audit aimed for was an assumption I'd never checked.
The score read 35, and the audit blamed how the scorer counted links. When I opened the scorer's code I found that problem and three more. Nearly half its points went to a style of link my repo isn't built around, and more went to a field for each doc's owner that I never fill in. It also had no limit on which files it scored, and it crashed on any doc filed under more than one area.
Because of that crash, the 35 came from a run inside another script, and I don't know what that script changed. I never logged July's sub-scores either, so I can't say how much of the 35 each defect cost. I ran a copy in October patched only to get past the crash, and the points for that link style and the owner field came out at about zero. So its number was pretty much how many files had metadata and how recently they'd changed, across files it should never have scored.
I left the old scorer alone and wrote a new one that scored links the way my docs point to each other. Its first run read 54. Most of that jump was the new scorer, since the two readings scored different files by different rules, and the rest was metadata I'd filled in between them.
That first run already gave metadata full marks and lost its points on links and starting pages. Only about one doc in four could be reached through any link, and only 5 of the 14 areas had a marked starting page. I added links and starting pages the next day and cleaned up the doc types. That took the score to 99 on the docs it covered in July.
The plan I wrote after the audit was off on its counts too. It had 731 of 2,836 docs with no metadata, off a quick scan that left folders out. Nobody wrote down how that scan worked. A scan held to just the docs that should be scored found 220 of 1,814 with none.
Two scores I built for the client ran short of data
The first was a job-posting scorer that tiered companies by how they were hiring, and the other was the account score I built about two months later. The job-posting scorer's live run in late July put no company in the top tier. One of its signals needed postings at least 31 days old, and collection kept only the last 7 days.
That signal didn't fire for a single company in the run, and another signal was dead too. With both dead, the best score its own weights allowed was 75, against a cutoff of 80. No company could have made it.
A rebuild about two weeks later let the dead signals fire, and the scorer's next live run still put no company in the top tier. The tier also needed one sub-score at 70 or more, and a review that same day found I'd never built the step that scaled it.
I put the account score's ranked list next to the client's real customers in early September. The top of the list came out 36 times the size of the median customer, because the score's heaviest signal came from data only large companies produce. I made size part of the score, and the top came down, though not to the size of a typical customer.
Its top tier held about a thousand accounts later that month, and they all scored between 70.0 and 75.5. Three of the things it scores had no data on any of those accounts, so each sat in the middle of the scale, the way the scoring recipe I published handles missing data. Together those three carried 40% of the weight. When data on one of them came in a week later, the same accounts under the same version spread up to 85.5.
Before I'd believe any change to the account score, I wanted the version in use to give the same answers twice, so I had it run again over every score it had already stored. All but 3 came back with the same score and tier, each of the 3 with a known cause. In the next version, I set the run to stop if fewer than 99% of the stored scores reproduce.
I wrote the rules for switching off my session digests before I measured them
Every working session I had with my agents was supposed to end with a digest of it, written in the background, along with a few content ideas. Both came out of the setup I built to capture every session. What got me measuring them was an agent that came back with nothing on a topic I'd spent weeks on, when 17 archived sessions had that topic in the conversation itself.
The digests turned out almost two usable items per session. A fresh agent checked 40 sampled items and agreed with 82.5% of them. On what they recorded, the digests looked pretty healthy.
But agents that never saw a digest read 18 raw sessions and listed the lessons in them. Once I weighted that across every session, the digests had missed 76.5% of the lessons and the content ideas had missed 91.0%.
I froze all 614 digests first, because old ones got rewritten. I signed off on the rules and locked them ahead of the measurements that decided the verdict, and the run log shows the order. The rules were to retire the digests if over 70% of lessons went missing or they turned out fewer than 5 usable items per 100 sessions, to leave them alone only past a much higher bar, and to fix them otherwise. The lock kept the verdict script from running on an edited copy.
In sessions that did get a digest, the miss rate was 67%, just under the 70% line. The sessions with no digest, a third of the total, pushed the overall rate over. I used no override, so the digests and the content ideas both got a retire verdict.
The verdict came back on September 27, and by itself it switched nothing off. I turned the digest worker off on October 13, about two weeks later, and session transcripts kept being saved.
The tests I ran on these numbers
I didn't need anything but the scorer in front of me to read its code, work out what its weights allowed or have it reproduce its own stored scores. For the others I needed something from outside the scorer to hold its number against, like my own past picks or the client's real customers.
Where each one stood in late October
| Number | Where it stood |
|---|---|
| The novelty score | As of October 20, every article still gets a novelty score and neither selector scores with it. The test on my past picks was a single run in one working session, with no script ever committed. The rebuilt selector's plan paused on October 4, and it never shipped an approved issue. |
| My repo's health score | I re-ran the new scorer on October 22 and it read 55, across over four times as many docs as its July runs covered. The old scorer, run on its own, still crashes on this repo. The setup that indexes my docs for search still has a setting that leaves out the files the starting pages live in, and no check on a doc's metadata runs when changes go into the repo. |
| The job-posting scorer | Its August 8 run put no company in the top tier. |
| The account score | As of October 22, the client still ran the same version of the account score. The next version, stop rule included, was waiting on a person's sign-off, with no change since October 8. |
| My session digests | I measured them in a single run. Between October 15 and 22, the step that runs at the end of every session fired 51 times and never started the digest worker. |
