Youness Aamiri

You add [MemoryDiagnoser] and [Benchmark] to a class, run dotnet run -c
Release, and BenchmarkDotNet hands you back a beautifully formatted table:
mean, median, standard deviation, allocated bytes per operation. It's one of
the best measurement tools in any ecosystem.

And then what?

You read the numbers. You compare them, in your head, to what you remember
from last week. Maybe you screenshot the table and paste it into a Slack
message. Maybe you don't, because nothing regressed today, so there's
nothing to say.

That's the gap. BenchmarkDotNet is a measurement tool. It has no opinion
about whether 4.5ms is acceptable or a regression. It doesn't remember what
the number was yesterday. It doesn't fail your build. It doesn't stop a pull
request. Every one of those decisions is left as an exercise for the reader
— which in practice means it doesn't happen, until someone notices
production got slower and goes spelunking through git blame.

A budget is not a measurement

The distinction I kept running into while working on
CedarRecon, a reconciliation
engine where I'm doing a lot of comparative benchmarking (dictionary-based
vs. indexed classification strategies, that kind of thing): a measurement
tells you what happened. A budget is a policy — a committed, reviewable
statement that says "this operation must not take more than X, and here's
what we do if it does."

Those are different artifacts with different lifecycles. A measurement is
disposable — you take it, you look at it, you move on. A budget needs to be:

  • Committed — visible in a pull request diff, not a number someone remembers
  • Reviewable — a change to the budget is itself a decision someone signs off on, the same way you'd review a change to a config file
  • Enforced — a CI failure, not a Slack message someone might read

BenchmarkDotNet gives you excellent measurements. It was never trying to be
the second thing.

Why not just eyeball it in CI logs?

The honest answer is: you can, for a while, on a small project with one or
two people watching closely. It stops working the moment either of those
conditions changes.

I found this out concretely, not hypothetically. While dogfooding an early
build of a tool I've been writing specifically to close this gap (more on
that below), I ran a real benchmark suite from CedarRecon, deliberately
introduced a regression into one method to confirm the tool would catch it
— and it did, correctly, twice. But on the same run, an unrelated method I
hadn't touched showed a 28% regression at the largest input size.

If I'd been eyeballing a CI log, that's the kind of number that either gets
ignored ("probably just noise, I'm busy") or gets chased for an hour before
someone realizes the machine was probably running warm. What actually
happened: I reran just that benchmark in isolation, and it came back
clean — comfortably within a few percent of baseline, no regression at all.
It was noise. A shared CI runner, or a laptop that hadn't idled down between
runs, produced one bad sample.

That's not a knock on BenchmarkDotNet — its statistics are good. It's a
demonstration of why "look at the number and decide" doesn't scale as a
process, even when the underlying measurement tool is solid. You need
something that applies a consistent rule every time, and — just as
importantly — something that treats a single suspicious result as a
hypothesis to confirm, not a verdict to act on immediately.

What a performance contract actually needs

Turning "here are some numbers" into "here is an enforced budget" means a
few concrete things have to exist that BenchmarkDotNet doesn't try to
provide:

A committed baseline. Not a number in someone's head — a file in the
repo. JSON, reviewable, diffable. When a baseline changes, that's a
deliberate act someone did on purpose, visible in the PR.

A threshold policy. How much regression is acceptable before it's a
failure? 5%? 15%? Does a tiny absolute change (1ns → 1.2ns) count, even
though it's "20%"? These are decisions, and they should live in
configuration, not in someone's judgment call at 4pm on a Friday.

A pass/fail exit code. The single most useful thing a CI step can
produce. Not a report someone might read — a build that goes red.

No SaaS, no hosted database. This one's a deliberate design choice more
than a hard requirement, but it matters to me: baselines and policies as
plain repo-local JSON mean no external account, no network dependency to
evaluate a benchmark, and — critically — the same command produces the same
decision locally and in CI. Nothing hidden behind a dashboard only CI can
reach.

Where this is going

That's the shape of the tool I've been building —
Cedar.BenchmarkGate,
a local-first performance regression gate designed specifically for
BenchmarkDotNet output. It doesn't run your benchmarks. It doesn't replace
BenchmarkDotNet's statistics. It reads the JSON BenchmarkDotNet already
produces, compares it against a committed baseline under a policy you
control, and returns an exit code CI can act on — plus a Markdown summary
that explains why, not just that.

v0.1.0-alpha.1 is live now, built and dogfooded against CedarRecon's real
benchmark suite — including that 28% false alarm, caught and correctly
dismissed by rerunning rather than reacting.

Next up: the architecture — normalized benchmark identities, why baselines
and policies are deliberately separate documents, and the exit-code
contract that makes this usable as an actual CI gate and not just a fancier
report.