You baked the model into the image. One env var can silently un-bake it

On serverless GPU, cold start is image pull plus model load. The single biggest win is usually to stop downloading weights at boot: download them at build time so they ship inside the image and load from local disk instead.

The setup is two lines. Point the cache at a path inside the image, then fetch during the build:

ENV HF_HOME=/opt/huggingface
RUN python -c "from huggingface_hub import snapshot_download; snapshot_download('<repo>', revision='<pinned>')"

Enter fullscreen mode Exit fullscreen mode

snapshot_download writes into HF_HOME, that directory becomes an image layer, and at runtime the loader finds everything already on disk.

The failure that doesn't look like a failure

Here's the part worth knowing. The loader resolves weights through HF_HOME at runtime. So if anything in your runtime environment sets HF_HOME to a different path — a network volume, a mount, a leftover template variable — the loader looks there, finds an empty cache, and quietly falls back to downloading from the network.

Nothing errors. Your image still contains the weights, sitting in a directory nobody reads anymore. You just paid to bake them, and you're paying the download on every cold start anyway. The only symptom is that cold starts went back to what they were before you did the work, which is easy to blame on the platform.

How to make it hard to get wrong

Attach no volume at all. If the worker has no volume, there's no tempting path to point the cache at. In my setup the endpoints run with volume size zero for exactly this reason: everything the worker needs is in the image, so a volume is not an optimization, it's a way to reintroduce the network.

Verify at boot, not by feel. Watch a cold worker's log. If you see download progress for weights, the bake is being bypassed, full stop. A baked image reaching the network for weights is always a misconfiguration, never a normal path.

Treat the cache path as part of the build contract. The Dockerfile that baked the weights and the runtime env that reads them have to agree. That agreement is invisible in both files individually, so write it down where whoever edits the template env will see it.

Worth saying out loud

Baking has real costs: the image gets large, builds take much longer, and updating the model means rebuilding and redistributing the whole thing. If your weights are small or your traffic keeps workers warm, a volume or a runtime fetch may genuinely be the better trade. This is about the case where you already chose to bake — because then paying the build cost and the download cost is the worst of both.