Getting a model onto a phone means exporting it, and almost always quantizing it. Both of those change the numbers. Everyone knows that part.

What got me is that nothing tells you when they change too much. The export succeeds. The app builds. The model runs on device and hands back something that looks completely reasonable. And the thing you shipped is worse than the thing you evaluated, by some amount nobody measured, and you find out from a support ticket a month later. Or you don't find out.

I went looking for the standard answer, assuming I'd just missed it. There is one, sort of. Compare your offline predictions against your online ones, and against the post-serialization ones. Every deployment guide says some version of that. Not one of them turns it into something a build can fail on. It's written up as a thing you should do, which in practice means you do it once, the week before launch, while you're doing forty other things, and then never again.

There's a second gap and it fails the opposite way. Preprocessing gets written twice. Once in Python, for training. Once in Dart, for serving. Same normalization constants, same resize, same channel order, sitting in two files that nobody diffs. They agree the day you write them. Then someone changes the mean and std on the Python side six weeks later and the app just quietly starts feeding the model something it's never seen.

So I built the thing I wanted. It's called Fluttorch, it hit 1.0.0 this week, and honestly the release is the least interesting part of this post.

One document, three readers

The idea is small: the exporter writes a manifest next to the artifact, and nothing on the Dart side is allowed to restate what's in it.

Shapes, dtypes, preprocessing constants, labels, the weight hash. Written once. Three things read them, the code generator, the parity gate and the runtime, and none of them declares its own copy. If the Dart side can't restate it, the Dart side can't disagree with training.

The runtime also refuses to load an artifact whose content hash doesn't match what the manifest recorded. That sounded a bit paranoid when I wrote it. Then I re-exported and updated only half the pair, and got an artifact that satisfied every shape check and returned every number wrong, and I stopped thinking it was paranoid.

The generated side

fluttorch_gen is a build_runner builder. It turns the manifest into Dart sitting next to it, and you commit what it generates so CI can regenerate and diff. That diff is what catches a manifest that moved without anyone rebuilding.

final classifier = await Classifier.load(runtime, artifact: bytes);
final out = await classifier.run(image: ClassifierImage(pixels));
final scores = out.logits.values;   // Float32List, over the same memory

Enter fullscreen mode Exit fullscreen mode

Three things that used to be runtime surprises are compile errors now. Passing the wrong tensor, since each one is its own extension type. Passing them in the wrong order, since run takes named arguments. Reading an output that doesn't exist.

None of that costs anything at run time. An extension type is just the underlying tensor once it's compiled. Wrong length still gets caught, at construction, and it tells you which tensor:

TensorShapeException on "image": expected 3072 values, got 100

Enter fullscreen mode Exit fullscreen mode

The bit I actually care about

At export time you hand the exporter some representative inputs and it records what the model answered. Those answers replay in your test suite, against the quantized artifact, on the real backend.

test('the quantized model still agrees with the one we evaluated', () async {
  final goldens = await DirectoryGoldenBundle.open(
    'build/classifier/classifier.fluttorch.json',
  );
  final model = await runtime.load(
    artifact: await File('build/classifier/classifier.pte').readAsBytes(),
    manifest: goldens.manifest,
  );

  await expectParity(model, goldens: goldens);
});

Enter fullscreen mode Exit fullscreen mode

The tolerance comes from the recipe and the precision the manifest recorded, so an int8-dynamic model isn't judged against the bound a full-precision one answers to. You can pass your own, and you probably should if your activation ranges look nothing like the ones the defaults came from.

When it fails it looks like this:

FAIL  parity/case-3
      backend: xnnpack  quantization: int8-static
      output "load_mw"  max |Δ| 1.72  >  Tolerance(atol 0.1, rtol 0.1, cos ≥ 0.998)
        worst at [0]: 14.0210 vs 12.3000
        2 of 4 elements (50.0%) exceed the elementwise bound
      no layer attribution: backend "xnnpack" offers no activation taps

Enter fullscreen mode Exit fullscreen mode

Which tensor drifted, which bound broke, which element broke it, which backend produced the number. I put all four in because I kept hitting failures where I had three of them and was still guessing. That last line is there because per-layer attribution needs the backend to expose intermediate activations and xnnpack won't, so instead of silently giving you less, it says so.

The thing I got wrong

I measured the default tolerances instead of picking them, mostly out of stubbornness, and the measurement told me something I'd have got backwards.

The two-layer model drifts up to eight times further than the convolutional one. The simpler network. I stared at that for a while assuming I'd broken the harness.

It isn't complexity. The two-layer model's outputs land around 9.4, the other ends in a softmax, and relative error is measured against the output while the rounding happened on intermediates. Big outputs absorb the same underlying error inside a much smaller relative bound. Obvious once you see it, and I did not see it for an embarrassingly long time.

One bound in that table is still unmeasured and says so, because nothing in the suite exports int4 yet and I'd rather have a hole than a plausible-looking number.

There's also a model that ExecuTorch happily lowers and then can't execute, identically under its own Python runtime. I left it in the suite as a failing expectation rather than skipping it, so whenever upstream fixes it, my tests are what tell me.

What it doesn't do

It doesn't train anything, convert between formats, or serve inference. It takes something you exported with torch.export and makes the border between Python and Dart typed and checked, and that's the whole scope.

ExecuTorch was just the first backend, not the shape of the thing. ONNX Runtime and LiteRT sit behind the same seam, and nothing above it knows which one is running.

Where it is

fluttorch and fluttorch_gen are on pub.dev, Apache-2.0. The native bindings aren't published because pub.dev can't carry the native half, so you pull those from the repo.

If you ship models to phones I'd genuinely like to know how you handle this, especially if the answer is "I don't". That's the part I'm least confident I've got right, and I'd rather hear it now than after someone builds on it.