Six AI Reviewer Agents Signed Off on a $426,000 Mistake

The brief is done.

Pages of analysis on a client’s ad spend, every number pulled, every chart built. I run it through my own review system before it goes out the door. Six agents, each one checking a different thing. Data accuracy comes back 9.5 out of 10.

Claude came back with, “Zero discrepancies.”

The number at the top, the one the whole recommendation hangs on, says $2.57 million.

And I just sit there looking at it. Something about it is too fat. I’ve been staring at this client’s pipeline for weeks, and $2.57 million does not match the shape of the business in my head. It’s not a calculation. It’s a gut feeling. The same feeling you get when you hand the cashier a twenty and the change comes back wrong before you’ve even counted it.

So I say the single most expensive sentence in AI-assisted work. “We need to double-check the numbers.”

The brief was off by $426,000 on its most important number, and my review had just stamped it 9.5 out of 10.

Then the worse part landed.

I had let Claude crunch those numbers freehand. No calculator, no script, no specialized tool. Just “here’s the data, run the totals.” And then I let a second pile of Claude agents check the first pile’s work. Two layers of AI. Not one of them did the math from scratch. The only thing in the entire pipeline that ever recomputed the numbers was my gut, and my gut doesn’t scale.

The gap between a number reading correct and a number being correct is a whole different story.

Fluency Was Never the Problem. Nobody Made It Do the Math Twice

The review wasn’t broken. In fact, it did exactly what I built it to do. The AI read the finished brief and checked if the numbers agreed with each other.

They did.

Page four says $2.57 million, page eleven says $2.57 million, the chart says $2.57 million. Consistent. Tidy. Yet wrong.

And that’s the trap. A review like this can only check the numbers it can see. The agent never went back to the raw export and added the deals up itself. The mistake didn’t happen in the brief. It happened upstream, in a script that quietly returned a null, zeroed out a bucket, and got backfilled with a draft number somebody had typed by hand weeks earlier. A number that matched no saved file.

By the time it reached the review, it had showered, shaved, and put on a suit.

This is the part people skip.

The $426,000 was not a math error. No model did that sum wrong, because no model did that sum at all. The wrong number came from everything around the math, and the review never looks there.

Asking AI to read a number and asking it to recompute a number are two completely different jobs. The review only ever did the first.

Five inspectors can all sign off on the bridge because the blueprints agree with each other. Not one of them weighed the truck. Consistency is not correctness.

More Review Agents Means More Confidence in the Same Wrong Number

When I tell people a review missed a $426,000 error, the first thing out of their mouth is “add another reviewer.” More eyes, more safety. It’s the industry’s reflex. Every multi-agent framework on the market sells you the same fix, an eval agent whose entire job is to grade the other agents.

Stacking reviewer bots has a problem. They all read the same finished number. None of them go back to the source. So you don’t get a second opinion. You get the same opinion in a different font, said a little louder.

The research on this is not comforting. In the 2023 benchmark study that put AI judges on the map, swapping the order of two answers flipped the verdict about 35% of the time, and that was GPT-4, the strongest judge they tested at the time. Weaker models flipped more than half the time. Same two answers, same judge, different order, different winner. And in specialized fields like law, medicine, and finance, the same pattern shows up: graders reward claims that sound confident over ones they can actually trace to a source. They don’t say “I’m not sure.” They sign off, with confidence.

In a multi-agent pipeline it gets worse, because that first wrong number gets written into the shared memory as ground truth and then laundered through every layer of plausible reasoning after it, until a human finally sees a figure that’s been blessed at every step and questioned zero.

Adding agents doesn’t add a check. It adds confidence. Those are not the same thing, and confidence is the one thing a $426,000 error already has in spades.

A lawyer in California found this out with a $10,000 fine for not checking AI outputs.

I Built That Review to Catch Exactly This. It Waved It Through

The part that stung was personal. I didn’t get burned by trusting some out-of-the-box tool. I built the review system myself. Six agents, hand-tuned, the thing I was proud of, the thing I’d tell other people to copy. Data accuracy was one of the six. Its entire reason to exist was to catch a wrong number. It looked at this one and gave it a 9.5 and the words “zero discrepancies.”

I felt the hit in my chest. It’s one thing for AI to fumble. It’s another when the safety net you stitched yourself turns out to have a hole shaped exactly like the thing falling through it.

And I don’t love admitting the rest. The real root cause wasn’t the review. It was me. I let Claude crunch the original numbers with nothing but a prompt. No spreadsheet, no script, no boring deterministic tool doing math the dumb reliable way. I asked a machine built to predict the next token to be my accountant, and then I asked more next-token machines to audit the accountant.

Everybody in the room was guessing. Confident guesses, beautifully formatted guesses, but guesses.

My gut caught it. A gut feeling is not a system. It catches exactly one number on one good day when you happened to be paying attention. Next brief, tired Friday afternoon, the gut’s asleep at the wheel and the $426,000 error ships.

The Cheapest Insurance You’ll Ever Buy

So I built the boring thing I should have had from the start. Not another reviewer. A Truth Gate.

A Truth Gate is a cold check between your AI and the person who’s going to trust it. Cold, as in run from scratch, blind to how the AI got its answer. It goes to the source, not the document. And it does not let anything ship until the two agree.

The idea is stupid simple. It’s the difference between proofreading a receipt and re-ringing the groceries. A review proofreads. A Truth Gate re-rings every item.

It’s one short file. It takes every headline number a reader would actually quote, the ones lifted straight off the document, and goes back to the raw data to recompute each one from scratch. It checks what the document claims against what the data says. Any gap, it stops you cold. It never reads the polished brief. That’s the entire point. It does the one job six AI review bots couldn’t, because they were all too busy reading.

Now the reframe that matters. That gate is insurance. The premium is ten minutes and one boring file. The payout is $426,000.

Nobody wants flashy insurance. You want it dumb, cheap, and sitting there the one day a six-figure error tries to walk out the door. If a Truth Gate looks flimsy to you, good. So does every insurance policy, right up until the house burns down.

So I put it on GitHub. One file, free, at github.com/biorevgrowth/truthgate. You pull it into any project folder with one line, and you never write a word of it yourself. You tell your AI to set up the truth gate for your report, it writes the checks from your data, and you run one command. The gate does the math. The AI just fills in the form.

There’s an obvious worry buried in that. If the same AI that wrote the report also writes the check, isn’t it grading its own homework? No. The independence doesn’t come from who writes the check. It comes from where the check looks. The gate never reads the AI’s document. It reads the raw data and recomputes blind, so the check runs on different inputs than the report did. The one thing you do is read the recipe once, just long enough to confirm it pulls from the raw export and not from the AI’s own math. After that it runs the same way every time, whether you are watching or not.

Watch it work on a pretend Q3 brief, fake data, no client numbers, where the whole headline rests on one figure, the late-stage pipeline, the kind that gets hand-keyed once and never checked again. It comes back with this:

  [OK] Q3 closed revenue: computed=331,500 claimed=331,500 diff=0
  [OK] Q3 closed count: computed=5 claimed=5 diff=0
  [OK] Q3 Enterprise closed revenue: computed=308,000 claimed=308,000 diff=0
  [MISMATCH] Q3 Enterprise late-stage pipeline (anchor): computed=435,000 claimed=461,000 diff=26,000

Total checks: 4   Passed: 3   Failed: 1

DO NOT SHIP. The brief disagrees with the raw data:
  - Q3 Enterprise late-stage pipeline (anchor): brief says 461,000, raw data says 435,000 (off by 26,000)

Three numbers check out. The anchor doesn’t. The script went into the raw deals, found the four that actually belong in that bucket, added them up itself, and got 435,000. The brief said 461,000. It didn’t take the document’s word. Exit code 1. In a real pipeline, that’s the send button frozen until somebody explains the gap. Fix the number to what the data says, run it again, and it goes green. “Clear to ship.” That’s the whole policy.

Run that same gate against my actual brief and it cold-checked all 45 headline numbers in about two seconds and flagged the one that was wrong by $426,000. The six-agent review, the smart one, the one that reasons and explains itself in full paragraphs, scored that same brief 9.5 out of 10. The dumb gate caught what the brilliant system waved through. That’s the whole bet. Boring beats brilliant when there’s money on the line.

Trust the output if you want. Then go cold-check it against the source, because the only number worth signing your name to is the one you counted twice.

One Gate, Everything AI Touches

This is where it stops being about revenue briefs.

A Truth Gate is a move, not a numbers tool. Anywhere AI hands you something a person is going to act on, you can drop a cold check between the output and the source.

Numbers, you recompute from the raw export. Citations, you re-look-up every case at the docket, the exact thing those lawyers needed and didn’t have. Code it wrote, you re-run the test on real input instead of trusting that it looks right. A date, a quote, a stat, you re-fetch it from the source document. Same move every time. Go to the ground truth, not the polished version, with a check that does not care how confident the AI sounded.

And it compounds. One $426,000 mistake, encoded once, never ships again. Catch a bad pipeline total and the cold check for it lives there forever, checking itself every quarter while you sleep. Catch a fabricated citation and the lookup is automatic from then on. Month by month your coverage grows. Year one, you’ve got a verification layer around your AI work that almost nobody else bothers to build, because the whole instinct points the other way, toward more reviewers reading the same output.

One rule, because I’ve broken it. Keep each gate to one file. The second it grows config flags and edge cases and a settings menu, it dies, because nobody maintains the clever version, including the person who built it. Small is also what lets you read the recipe and trust it in the first place. A Truth Gate earns its keep by being small enough to eyeball and boring enough to actually keep.

That’s the edge. Not that checking your numbers is new. Finance has tied out figures for a century, and accountants built a whole profession on it. The edge is that almost nobody does it to the AI output they actually ship, and the reflex everyone follows points the wrong way. The wall isn’t hard to build. It’s that almost nobody thinks to.

The Lawyers Who Got Fined Were Uninsured

This is what going without it costs.

Start with that California lawyer. In 2025 he was fined $10,000 for an appeal where 21 of 23 quotes were fabricated by ChatGPT. Then he did the thing that should sound familiar. He ran the brief back through Claude, then Gemini, then Grok, hoping one of them would catch the fakes. They amplified them. He admitted to the court he never read the output before he filed it. Two firms got hit for $31,000 on the same kind of thing. An Oregon case, $15,500, fifteen fake citations and eight invented quotes across three briefs.

Every single time, the judge made the same point. They weren’t punished because the AI made things up. They were punished because nobody checked. The machine’s mistake was free. The failure to verify is what landed the fine.

That’s my $426,000, wearing a robe. A lawyer trusting quotes he never re-read is the same animal as six review agents passing a number none of them recomputed. None of them had a Truth Gate. One cold check against the docket, one lookup, and every one of those fines goes away.

This is anyone shipping something somebody else is going to trust. The agency emailing a client a monthly ROAS report. The consultant with a market-size number holding up a whole strategy deck. The founder pasting MRR and burn rate into a board update at midnight. The analyst whose model feeds the forecast everyone plans against. If a claim in your deliverable is load-bearing, and an AI touched it, and it’s going to someone who’ll act on it, you are uninsured against your own $426,000.

Start Here. Audit What You’re Not Insuring

Before you build a single gate, find out where you’re exposed.

Run that against your own operation. Every empty box is a number, a citation, a line of code, or a claim that an AI produced and nobody checked against the source. That is your uninsured exposure, and it is sitting in deliverables you have already sent.

Then buy the cheapest policy first. Pick the row that scares you most, usually the numbers.

You can write the ten-line version yourself, or pull mine off GitHub and tell your AI to set it up. Either way, on your next deliverable, before you hit send, run it against the raw data. If the numbers match, ship. If they don’t, you just caught a $426,000 before it caught you.

One file. Three numbers. Ten minutes. You don’t need to be a developer. I’m not one. The math the gate does is the same math you’d do by hand if you had the patience, which you don’t, which is exactly why it exists.

Then add the number that burned you. The gate gets smarter. Your coverage grows. You get embarrassed less.

The question was never whether your AI can run the numbers.

It’s whether you’re covered the day it runs them wrong.