Ten days ago I had a benchmark: reorder the expert weights inside a MoE model file by measured co-activation, and disk reads per token drop 2.23×. It was real, reproducible, and measured entirely on my own machine — which is to say, it was worth almost nothing yet.

Today the strongest number attached to this work is one I didn't produce: +32.3% decode throughput and −26.3% time-to-first-token on a 235B-parameter model running on a 48 GB MacBook, measured by someone I've never met, on an inference engine I'd never run, using my script unmodified, with the arms swapped as a page-cache control. Along the way, two of my three original pitches were refuted with data, my one pre-registered scaling prediction failed its own threshold, and both of my attempts to forecast a key parameter missed in opposite directions.

The refutations were the productive part. This post is about the process that made them cheap: five engines, a half-dozen strangers, one issue thread, and the discipline that emerged in it — because I think the process generalizes and the write-ups I usually see don't mention it.

The setup

The idea (project: mbolt) is BOLT/PGO applied to model binaries. Checkpoint tensor order is an accident of the training pipeline; MoE routing at inference time is far from random — experts fire in cliques. If an engine streams experts from SSD, the file layout decides whether a token's misses are a handful of long sequential reads or thousands of scattered ones. So: trace the routing, cluster the co-activation, rewrite the file, keep the weights byte-exact.

After a week solo I had the 2.23× read reduction on an 80B model, a 1.55× end-to-end decode gain against a patched llama.cpp, correctness gates (byte-identical weights, routing mapping 100.000% through the permutation, output divergence 5× below the same engine's CPU↔Metal backend delta), and a 500-prompt blind A/B that scored 48 to 46 with 26 ties — a coin flip, which for a layout change is the goal.

I also had a null I'd measured myself: on stock llama.cpp, none of it matters. The mmap page-fault path is layout-blind; I measured parity and said so. The gains need an engine that issues explicit reads.

That null dictated the launch strategy, although I didn't think of it as strategy at the time. The people who run explicit-read expert streamers are countable. Instead of posting a Show HN, I took the data to their issue trackers: ml-explore/mlx-lm#1438, where @mabaeyens was shipping an expert-offload streamer for MLX, and JustVugg/colibri#119, a C engine that streams a 744B model from NVMe. Opening line: sharing data because your engine is the natural home for a layout pass — and one of my findings says the audience for this is you, specifically.

First contact: two pitches die in one reply

mabaeyens did not argue with my framing. He measured it.

His reply took my three pitches — pin the hottest experts in RAM, prefetch the next layer's experts by co-activation, and reorder the file — and killed the first two with his own engine's counters. Heat-pinning a resident set loses to a plain per-layer LRU even on same-topic, same-session traffic: the LRU already banks the reuse, and every frozen slot is a slot the workload's drift can't use. And cross-layer prefetch has no signal to run on: the expert overlap between adjacent layers measured 0.017 against 0.016 for a shuffled control. Independent per-layer routers plus the load-balance loss decorrelate the layers. There is nothing to predict with.

I retracted both, in the thread, that day.

Here is the thing I want to highlight, because it's the hinge of the whole story: the thread treated the refutations as scoping, not as defeat. The claim didn't die; it moved. If retention kills the decode win, the layout lever migrates to where reads still happen — cold prefill, and (as his next measurement showed) decode too, once the expert table is so much bigger than RAM that misses stay cold forever. His fragmentation numbers made the case sharper than my original pitch: his reader issued ~471 file opens per decode token, nine byte-ranges per cold expert. The lever wasn't dead. It had been mislabeled.

The rules the thread converged on

Nobody declared a methodology. It accreted, one burned hand at a time, and by the end it looked like this:

Lock the experiment before running it. When we designed the decisive test — coalescing alone (profile-free, deterministic) vs coalescing plus co-activation reorder (needs a trace) — the arms, metrics, and decision rule were agreed in the thread before anyone ran anything, because the answer determined whether he'd break his engine's read-straight-from-shards invariant.

Pre-register numbers, with thresholds that can hurt you. My on-record prediction: at deeper offload the coalesce gain reaches ~+20%, and "if it lands under +15%, the honest write-up says this lever tops out at nice-not-necessary." It landed at +9.7%. I conceded in full — no appeal to config differences, because the pre-registration had already forfeited them.

Concede at commit speed. My analytical model over-credited a 36 µs open() cost his reader had already eliminated — his shared-fd fix landed about four minutes after he quoted the number I'd built on. When his measurements came back lower than my model, the model lost, publicly, same day. Score-keeping note: across the thread I went 0-for-2 forecasting the cold-read share (missed in opposite directions both times) and the community's rule hardened into measure it per engine, never carry it across.

Controls, or it didn't happen. The thread's canaries are my favorite artifact. A "2.23× speedup" that was actually macOS refusing to evict resident pages (F_NOCACHE does not do what three of us assumed; the tell was an impossible 0.18× in the control arm). A prefetch A/B that "confirmed" the predictive arm until its author discovered both cells had run the same code — byte-identical hit rates were the tell — and re-ran it properly: the predictor lost by 2.9%, reading 6.9 bytes per byte it saved. A cross-architecture overlap claim that showed 1.17× signal against an analytic uniform baseline and 0.98× against a shuffle control — the analytic baseline was manufacturing the signal. Arm-swapping for page-cache state became standard for every A/B in the thread.

Name the baseline or the number doesn't transfer. The same rewrite is +32% against a reader that issues nine scattered reads per expert, +31% against three-per-projection at deep offload, and ~nil against a container that already stores each expert contiguously. Quoting a layout gain without naming the baseline read pattern is how numbers stop replicating.

Maintain one living artifact. Midway through, I posted a consolidated summary as a comment and kept it versioned — v1.0 through v1.5.1 so far — edited in place via the API, with a changelog, and explicitly red-line-able by participants. People used it. One participant's red-line was purely attributional (the engine's name — which I then managed to misspell for two days; corrected in v1.4.1, and yes, that's in the changelog). The summary, not any single result, is what the thread produces: scoped findings, the measured points, and a traps section.

The falsification that became the payoff

The +9.7% failure was the best thing that happened to the project.

The mechanism behind it — coalescing is an IOPS lever, and a 4 MB slice is already bandwidth-bound — generalized into what the thread now calls the slice-size law: gain ≈ 1 + saved_ops · t_op / (slice_bytes / BW), with t_op ≈ 100 µs and BW ≈ 6 GB/s on this hardware class. End-to-end, it composes with the cold-read share s as speedup = 1/((1−s) + s/g). It is an engineering fit, not physics — but it now has six measured points across five engines, and it does the one thing a good model does: it tells you in advance whether your configuration is worth the trouble.

The law's test came from the two participants who joined after the initial exchange. @lBroth ran my conversion script, unmodified, against a 119 GB Qwen3-235B build and wired the output into his own engine: +32.3% decode — not because the per-read gain was large (it wasn't; big slices, g ≈ 1.34) but because at 2.6× over-DRAM the cold-read share is ~0.96 and the law's other term takes over. Then he built the deeper variant inside his engine and measured +31.2% where the law, priced against his named baseline, predicted +31.3%. And @pierre427 — running a production GLM-5.2 pager — pre-registered the per-read gain band we derived (1.2, band 1.15–1.3) and measured 1.195. The law's first registered-then-measured point.

The other end got priced too, and this matters just as much: @philipjohnbasile measured his 744B engine's container — which already stores each expert as one contiguous ~19 MB read — at a residual +1.3%, below his own promotion gate. He posted it as a no-go, I agreed, and the ~+20% GLM-class number I'd floated earlier was retired. The coalescing ladder is now fully priced: nine-scattered → +14–32% depending on share; per-projection → +31% at deep offload; per-expert-contiguous container → approximately nothing, at device bandwidth ceiling. An optimization whose domain of inapplicability is measured and published is worth more than one with only success stories.

Nulls are product

The thread's graveyard, as of today: predictive prefetch (three independent nulls — the structural one, the properly-controlled A/B, the cross-architecture shuffle test), heat-pinning versus LRU, smarter-than-LRU eviction, and my original cross-layer framing (retracted). Each null is scoped — batch-1, decode, these architectures — and each one is a roadmap item somebody now doesn't have to build.

Then the ending wrote itself. Late in the thread, PhilipJohnBasile open-sourced his engine — and its README turned out to contain an independent replication of nearly the entire design theory, measured on a fifth engine before we'd ever spoken: router-lookahead prefetch reverted at −6.96%, smarter eviction dead at −26% with zero hit-rate change, static pinning dead, hotset training dead. His summary line: "the cache-policy frontier is closed: capacity vs reuse distance is the wall." We had converged on the same wall from four different codebases.

(That open-sourcing also resolved a citation mystery in one of the great small moments of the thread: a throughput figure everyone kept attributing to our measurements turned out to be an Italian code comment at line 6141 of the parent project his engine forks — measured on Linux in early July, a platform qualifier his fork's copy had dropped. Two of us had "verified" the wrong provenance because GitHub's code search silently doesn't index files that large. That's in the traps section now too.)

What I took away

  • Take claims to the people with the machines to kill them. Four engine authors falsified, scoped, or extended every major claim within days. No general audience does that. The niche thread was not the humble alternative to a launch; it was the higher-bandwidth channel.
  • Pre-registration is what makes being wrong cheap. Every number I committed to in advance either survived (and now carries weight) or died cleanly (and produced a law). The two forecasts I hedged on are the ones that cost follow-up rounds to untangle.
  • Public concession is the currency. The thread's author said it at the close, better than I can: "I learned more from the parts where I was wrong than the parts where I was right. Good thread."
  • Keep one versioned, red-line-able summary. It outperforms both the scrollback and any paper: it's current, it's editable, and every participant has standing to correct it.
  • Ship your nulls with your wins. The parity result on stock llama.cpp, posted up front, is why the right people engaged at all.

The thread is still open. Co-activation-aware ordering is now the named follow-up on the mlx-lm PR; the colibri community is building an expert atlas with the co-activation format from this work; there's a co-signed, pre-declared cross-engine experiment waiting on two routing traces. The consolidated summary is here (pinned near the end of the thread), the reproducible pipeline is public, and the repo is github.com/doramirdor/mbolt.

If you have a benchmark you believe in, find the five people whose engines can falsify it, and post it where they live. The worst case is you lose a claim. The best case is you lose two and gain a law.

Credits, in measurement order: @mabaeyens (mlx-lm expert offloading, and most of the refutations), @lBroth (scale validation, in-engine builds), @philipjohnbasile (the 744B endpoint, iliria), @pierre427 (production pager, the registered-band point), and the colibri crew — @JustVugg, @ZacharyZcR, @bokiko, @mohamedmastouri2000-boop — who are taking the co-activation work somewhere none of us planned.

For more information and other projects go to https://getnadir.com