Why we needed AI to read a thousand pricing pages

This summer we set out to learn how software companies actually charge for AI, from real pricing pages with real numbers, across about a thousand AI and software companies. It was going to feed a private memo for a client, a public white paper, and a public pricing guide planned for this site.

I ran it the way I run most of my work now, with AI coding agents building and running it from plans I directed. An agent is a program that uses an AI model to work through a multi-step job on its own, and a model is the kind of AI that sits behind ChatGPT.

Nobody reads a thousand pricing pages by hand, which is why this was an AI project. Agents downloaded each company's pricing pages. Then a model was supposed to read every page and pull out the prices and plans in the same shape every time, so a thousand companies could sit side by side in one table.

Before you trust a model with a thousand companies, you try it on a few dozen and grade its answers against ones you already know are right. We ran that test on 40 companies and it failed, which is pretty normal for a first test. Then we went back through what had actually run and found that none of the pages had been read.

Here's where I landed. You need a direction, clear questions you want the data to answer, and a bulletproof, tightly scoped plan. Then you need small data sets and bite-size pieces that a person actually opens and looks at, because even careful AI, with evaluators and independent checks stacked on top, can go down the wrong rabbit hole. We did, several times.

No check on our 40-company test looked at what had actually run

The test launched three AI worker agents, each running a cheaper model and each handed its own batch of the 40 companies. Their job was to read each company's pages with that model and fill in the answers, and that reading never happened. Each batch's results were written by a small script, which here means ordinary code that does exactly what it's told, with no AI involved. Two of the scripts were lookup tables with the answers typed in ahead of time, and the third was a chain of rules that amounted to "if it's this company, write this."

Our records keep each script's file name, a fingerprint (a short code computed from the file's exact contents) and the exact lines where the typed-in answers sat. They don't keep the scripts themselves, and they don't say where the typed-in answers came from or who wrote the scripts. My best guess, and it is a guess, is the three workers themselves, since there was one script per batch.

Every result still carried a label naming the AI model that supposedly wrote it. Each one also reported zero tokens, which are the units AI companies use to meter how much a model reads and writes, and a model reading even one pricing page uses plenty of them. The record of which model ran and what it cost was blank on all 40.

Every result also had an empty list where the prices and plans should have been. What the scripts did produce was 46 small factual statements about the 40 companies, and 17 companies got none.

None of our checks caught it, because each one was checking something else.

  • The integrity check confirmed the run finished without crashing. It had no way to know who wrote the content.
  • The answer form required every result to fit a set shape, and an empty answer fit.
  • The grading compared the results with an answer key. That's the job of an evaluator, which is a program or model that scores answers against ones you already know. Of the 46 statements, 40 held up when checked against the pages, but of 91 answers we already knew were right, the results found only 20. Neither number could tell you whether a model or a script had written them.

Before any of this, we had written down twelve core questions the finished data needed to answer, and the test answered none of them. Nothing in the scores pointed at scripts. We found those by going back through what ran, which files wrote which results and what was typed inside them.

I've seen this before, in a system I built for drafting articles that reported steps as finished when they'd never run. This time it showed up in a data project.

Proving every answer came from a real model took two weeks and finished zero companies

So we built proof into every step.

  • Empty answers stopped counting. A blank result now got rejected.
  • Every result got a stamp. It recorded which program wrote the result and which one reviewed it, in a way that showed if anything changed later. Our notes admitted the limit. The stamp is us vouching for ourselves, and nothing in it comes from the AI company confirming its model ran.
  • Zero tokens meant no model. A result reporting zero tokens and a blank record could only count as ordinary code. That closed the exact hole the fake test used, though a result whose record had a model's name filled in could still report zero tokens and pass.
  • A fair test, planned and never run. We designed a cleaner trial. Practice on the original 40, then check against 60 fresh companies with at most three tries to look at the scores and adjust, then take one final shot at 80 companies nobody had seen, the way a final exam uses questions you didn't practice on. The 60 and the 80 never ran.

Meanwhile the code that ran each group of companies kept tripping over details that had nothing to do with pricing.

  • One version died before any model was called. Two parts of the code stored the same text with different invisible characters, so a check decided the file had changed.
  • The first company to make it all the way through the run was refused at the end. One page showed a $0 price with no region (which market the price applies to), the rules required one, and no price from that company was accepted.
  • Three more companies produced good results, and all three were set aside after the model finished. Our check flagged their files as changed. Some of that was real, because the code had moved on in the meantime, and some of it was the same invisible-character problem, this time from Windows and Linux ending lines of text differently.

On July 30, four working days in, we counted where things stood. We had pages downloaded for 981 companies, a model had worked on ten of them, and not one company had made it all the way through. By August 8, when we stopped, still no company had been accepted end to end.

The whole stretch ran from July 27 to August 8, two weeks on the calendar but about six days of actual work, with a quiet week in the middle. A count in October put the code written for this approach at about 129,000 lines.

Looking back, I think the direction drifted. We started out asking how companies price AI, and somewhere in those two weeks the question we were really working on became whether our own setup could be trusted. Our build notes from that stretch say writing new code for each small group of companies was hiding that we weren't getting through the actual list. I don't think the plan ever said how much proof was enough for the whole effort.

Small batches of nine got us through all 978 companies in two days, ungraded

On August 8 we set the proof work aside and switched to something much plainer. Agents read the downloaded pages nine companies at a time and filled in the same fixed form for every company.

That loop ran 110 rounds of up to nine over August 8 and 9 and covered 978 companies. Three more were left out because their source pages had problems.

When a page didn't say something, the record said unknown, and that happened a lot, 6,548 unknown entries across all the form's fields against 2,322 actual prices. On one company an apparent $60 million figure turned out to be financing, so it stayed out of the price column.

These records have the same hole the fake test had. The agents did the reading inside their own working sessions, and that usage was never tied to any one record, so none of the 978 shows which model wrote it or how many tokens it used. Our run notes say a model did the reading, because that's how we ran the agents, and that is the same kind of evidence that fooled us at 40.

We also haven't shown they're right. Our records show no scoring of the 978 against answers we already knew.

Batches of nine were small enough for a person to open, and our records don't show anyone opening them. Five companies even ended up with two records each, kept on purpose so someone could compare them, and nothing shows those were compared either. (I've written before about agents falling apart when you hand them too much at once, and this was the same lesson at the size of a whole project.)

Opening ten "captured" screenshots before the big paid run found seven error pages and forms

This is the part that worked the way it should. Alongside the pricing data, we saved a screenshot of each company's pricing page as a record of what it said on the day we looked. Automated capture will report "captured" on an error page, so the plan required a look at ten test screenshots, by a person or an agent, before the big paid capture run. Nothing bigger could start until that look was done, and an agent did it.

All ten came back marked captured. The files were big enough and they were valid images. A check on the page text did flag three of the bad ones, but it only warned, so all ten still counted.

Then the agent opened each picture. Three of the ten were pricing pages. The other seven were four "page not found" errors and three forms asking you to book a demo. One of the errors was Snowflake's, and it read "PAWS RIGHT THERE!"

The look stayed part of the process after that, and it kept catching things. In later cleanup work, every screenshot was reviewed before it counted, though our records don't say whether a person or an agent did the reviewing. Of 72 screenshots that passed every machine check, 26 were rejected at review, about a third. Across the last 542 companies on the list, 61% of the screenshots that passed the machine checks were rejected.

The reports passed their release checks, and the writing was still rejected

The last wrong turn was in the writing. The white paper came out as a 21-page PDF with each of its 59 claims traced back to a source, and the private client memo draft ran to 6,722 words. Both passed their release checks, the automated checks a draft had to clear before it counted as done.

Some of those checks were genuinely good. One scanned all 85 files in the white paper's public release for private client material and found none.

The prose was rejected anyway. An editorial note an agent wrote afterward called the drafts "safe but often cognitively punishing, overqualified, internally focused, and strangely bloodless."

The plan we're running now reuses the August data and starts the writing over. Someone has to read the new drafts themselves, the way the agent opened the screenshots, which I've written about from the writing side.

How to check AI agent output before you trust it

If I were starting this project over, I'd do three things, in this order.

Write it all down before you build. One sentence of direction, the questions the data has to answer, and a plan that says what done looks like. Our twelve questions are why nobody could argue the first test had worked, since it answered none of them. We had no finish line that sharp for the proof work.

Work in small batches, and keep some answers back. Nine companies at a time is what finally got us through the list. Hold back a set of answers you know are right, and keep them away from whoever, person or agent, is building the thing, so you can still score it fairly at the end. The only answer key we had covered 15 companies, and it had been opened and copied into readable files, so it no longer counted as a fair test.

Look at the thing itself, a handful at a time, every batch. Open the screenshot or the record and see what's in it. It can be you, or an agent you've told to open the actual files. The zero tokens were in the fake test's records the whole time, and since the plan that ran the test listed zero model calls and zero spend as part of a clean pass, we read them as good news.

Of everything here, the screenshot check is what I'd copy. Next time I'd give the drafts the same kind of check.