Every team I've watched ship a broken measurement system broke it the same way. Not with bad math — with an org chart problem that happened to live in code.

The entity making the claim ended up being the entity that decided whether the claim was right.

Once you have the shape in your head you start seeing it everywhere.

Three roles, not two

Most engineers think about measurement as two roles: the thing that acts, and the thing that grades it. That's one role short. There are three:

  • Player — makes the claim. Your model, your service, your PR.
  • Scorer — applies the rubric. Your eval harness, your test suite, your metrics dashboard.
  • Settler — determines what actually happened. Production outcomes. Reality.

The scorer is a proxy. The settler is the thing the proxy is trying to approximate.

The rule: be the scorer, never the settler. When the player captures the settler, the loop closes on itself and the system can no longer be wrong — which sounds like success and is actually the failure.

What it looks like in code

Tuning on the test set. You check test accuracy, adjust hyperparameters, check again. Twenty iterations later the test set is training data with extra steps. The player is now selecting its own settler. That's what overfitting is, structurally — not a math failure, a role-collapse failure.

LLM-as-judge from the same family. Your generator is GPT-flavored and your judge is GPT-flavored. They share pretraining data, failure modes, and blind spots. The judge doesn't rate quality — it rates similarity to what it would have produced. Correlated error is invisible to averaging; running it 1,000 times makes you more confident of the same wrong answer.

Benchmark contamination. The model scores 94% on the benchmark that's in its training data. Nobody lied. The settler just quietly moved inside the player.

Self-reported health. A service that returns its own health check is a claimant ruling on its own claim. If the process is wedged, the check is wedged too, and your dashboard is green through the outage.

Backtests without decision-time reconstruction. You score a historical decision using a feature computed after the outcome was known. The past gets graded with information from the future. Your Sharpe ratio is beautiful and entirely fictional.

Same failure, five costumes: the claimant captured the settler.

The fix is structural, not moral

Nobody in the above list was dishonest. That's the point — this doesn't get solved with care or seniority, because it isn't a discipline problem. Independence has to be engineered or it decays under ordinary pressure.

What that means concretely:

Make the settler expensive to reach. A holdout you can query freely is not a holdout. Rate-limit it, log every access, treat a query as a spend. If the number of times you've looked at the test set isn't recorded anywhere, you've already lost track of how contaminated it is.

Pick judges from a different failure family. If a human can't review it, at minimum use a different model lineage. You want errors that are uncorrelated, not errors that agree with each other.

Let production be the settler. Ship the prediction, capture the realized outcome, compare. Anything upstream of reality is a proxy, and every proxy is a scorer wearing a settler's coat.

Reconstruct decision-time state. Before scoring a historical decision, rebuild exactly what was knowable at that moment. If that's hard to do, that difficulty is the finding — it means your current backtest is leaking and you didn't know.

Watch who owns the metric. If the team that ships the feature also defines success and reports the number, you don't have a measurement — you have a press release with error bars.

The check that catches it

One question, applied to any number your system produces:

If this claim were wrong, what would tell us — and is that thing under our control?

If nothing would tell you, the claim isn't false, it's unfalsifiable, which is worse. And if the thing that would tell you is something you own, tune, or can silence, you're the settler and the loop is closed.

A closed loop doesn't fail loudly. It drifts, confidently, until reality collects — and reality runs on its own clock, not your sprint schedule.

Why I keep thinking about it

I build software that tells medical practices which claims are worth appealing. The settler is the payer's actual remittance. Not our model's confidence, not our internal rubric, not what our dashboard says. If we ever let those become the settler, we'd have a product that looks excellent and is quietly lying to people who make payroll decisions with it.

I ended up writing this up properly — the same structure turns out to show up in market bubbles, in overfitting, and in a few places I didn't expect: The Uncapturable Judge.

Where's the closed loop in your stack? Mine was the health check, and it took an outage to find it.