I built a page that only publishes numbers surviving our own checks. It caught me inside the hour.

I run a verified inference endpoint. What we sell is checking: we verify results before serving them, and we say so when we cannot.

Every claim defect we shipped this month had the same shape. A number was measured honestly, the configuration underneath it changed, and the number stayed on the page. Our benchmark rows were measured on model tiers we retired weeks earlier. Nothing was fabricated. Everything was stale.

So I built a page that cannot go stale. It is generated from the live gateway config, and every badge on it is derived rather than written. If a measurement's configuration premise no longer matches what is running, the page strikes the number and says why.

It went live. Within the hour it caught me publishing two numbers our own ledger explicitly withholds.

What the page does

For each path through the system it states four things:

1. What "good" means on that path, and whether a real oracle checked it. Tool calls are checked against the schema. Code runs against your tests in a sandbox. Free prose gets nothing, because there is nothing to run.
2. The measurement, with n, interval and date.
3. Whether that measurement still describes the running system, graded automatically against live config.
4. Whether it passed adversarial review, and if not, that appears on the page next to the number.

The third item is the whole point. A number and a config are two facts that drift apart silently, and no amount of care fixes that. A generator that reads both cannot lie about the gap.

The catch

I put our tool-calling and trap-set results on it. Both are good numbers. Both are marked in our internal claims ledger as not cleared for external citation, owes adversarial verify.

I had written the page whose entire purpose is publishing only what survives our checks, and I had not run the check.

The first fix was a blocklist for that one phrase. Also wrong. Our ledger withholds rows in at least six phrasings, and a blocklist misses the seventh. For a public claim the gate has to fail closed: a figure publishes only if its ledger row exists and carries no hold marker. Over-withholding costs a number on a page. Under-withholding costs a retraction.

Failing closed immediately caught two more, subtler ones. Two figures had no ledger row at all, because I had sourced them from an internal handoff instead of the truth layer. One was a 30/40 result. Our ledger also contains a 30/40, from a completely different run on a different model. Same number, different measurement. That is exactly the confusion an uncited figure invites.

Then the reviewer rejected my draft

The remaining claim needed real adversarial review, so I commissioned one and briefed it to refute rather than agree. It came back do not publish, with three specific defects:

• I wrote that our agreement gate accepts a wrong answer "roughly a quarter to a third of the time." That range was two underpowered subgroups (6 of 25, and 5 of 15) whose confidence intervals almost entirely overlap. It reads like a stable operating characteristic. The data supports one pooled binomial: 11 of 40, 27.5%, Wilson 95% interval 16.1 to 42.8.
• I quoted "1.7 to 3.5% on code." It has the shape of a measured interval. The reviewer said the evidence was not in my brief, which would normally mean I under-briefed it. So I checked instead of accepting. The figure appears three times in our ledger with no n, no interval and no date anywhere. The gap was real.
• I wrote "near zero on arithmetic." That is residue from a claim we retracted publicly after a reader caught it from our own article's text. No clean replacement exists, so the phrase cannot appear at all.

I took all three. I refused one instruction: it wanted me to name an internal threshold, which is recipe rather than result. It did not know that because I had not briefed it on our disclosure policy. Same failure, other direction.

The green check that could not fail

While extending our claims gate to cover more surfaces, I did something I should have done when it was written: I injected the defect it exists to catch.

Seven injections. Five wrong percentages, a struck cost multiple, a retracted compounding claim. It fired on none of them. Its tolerance covers most of the range it checks, so almost any number is "backed" by something. It had been printing OK every morning for weeks.

It was not lying. Its own output says "values checked, not claims," and a comment in its source says the same. The defect is one level out: a green line on a status board is read as protection, and nobody re-reads the caveat.

A guard you have never watched fail is not a guard. It is a habit.

The rules that transfer

A correction in your repo is not a correction in public. Our live site was still serving retracted figures while every internal instrument reported the sweep as done, because every internal instrument greps the repo. Verify a public claim by fetching the public URL.

Your measurement tool is an instrument, so verify it first. Our cost script summed one of three cost columns. It reported one cent where the truth was $1.73, and it was the exact script our ledger's cost row cited.

A readability filter can eat an error. I piped a remote script through grep -v to hide a deprecation warning. A database auth failure exited zero, and that became STATUS: Success, then a fleet-wide OK across both production boxes. The lie got more authoritative at every hop.

Placeholders are claims. Our dashboard rendered "$1,284 saved this month" as static HTML before live data arrived. A brand-new account with zero usage would have been shown a fabricated saving. I found it while opening signup to the public, one step before strangers would have seen it.

Publish the path where you are weaker. Our trap set is 200 of 200 on the primary path. Under failover it is 30 of 40, and every miss is the same verdict: it calls a tool where it should decline. Both numbers are on the page. The second is the more useful one.

Go and check it

The page is live at tirtha.ai/capabilities. Seven paths, what checks each one, what it measured, and the review state of every figure.

To try it rather than read about it, tirtha.ai/verify runs live against our API with no key and no signup. Three prefilled adversarial cases, and you can watch it decline a call instead of inventing one.

To run it on your own workload: go to tirtha.ai, click Get an API key, sign in with Google, and start. Free while we are in beta. No card, no email, no sales call. You get 100 requests a month and it stops rather than surprises you.

If you find a claim on that page that does not survive, I want to hear it. That is not rhetorical. The last time someone did, they were reading our own published text, and they were right.