Natalie Yevtushyna

"Just add a fallback model" is the kind of advice that sounds complete until you've actually shipped it. A single backup model handles one failure mode (the primary is down) and quietly ignores the other four that show up in production: slow responses, malformed output, rate limits, and a fallback model that doesn't accept the same context shape as your primary.

Here's the pattern breakdown that actually holds up.

The five patterns, and when each one is the right call

Pattern What it does Best for Main risk
Retry Same model, again, after a delay Timeouts, brief 5xx, short rate-limit windows Too many retries just adds latency
Fallback routing Different model or endpoint Primary is down or unhealthy Output can differ meaningfully from the primary
Load balancing Spreads traffic across healthy paths High-volume traffic Behavior varies slightly by which path you land on
Caching Reuses a prior response Repeated FAQs, deterministic tasks Stale or wrong cache hits if governance is loose
Semantic routing Routes by meaning/intent, not just a static model name Agents, multi-domain apps Misclassification routes the request badly

None of these replace the others. A production system usually needs some combination: check cache, check model health, classify the request, pick a route, validate the output, log the decision. Skip the validation step and you'll eventually route a mangled response straight into a downstream tool call.

The part that actually bites: model context

This is the failure mode that doesn't show up until you've already shipped the "add a fallback" version. Model context, meaning system prompt, conversation history, retrieved documents, tool schemas, output format rules, isn't guaranteed to survive a switch between models cleanly. Things that quietly differ:

  • Context length limits
  • How system prompts get interpreted
  • Tool/function schema expectations
  • JSON formatting strictness
  • Streaming behavior

OpenAI-style and Anthropic-style APIs use genuinely different request shapes (OpenAI reference, Anthropic Messages API). An "OpenAI-compatible" gateway smooths over the request/response format, it does not guarantee the fallback model behaves the same way given the same input. That distinction matters and is easy to skip past when you're just trying to get failover shipped.

For agents specifically, this compounds

A single user task can trigger planning, tool selection, summarization, and a final response, each a separate model call. If any one of those fails without a sane retry or fallback path, the whole task can collapse. Worth building in from the start:

  • Retry limits with backoff and jitter (not unlimited retries)
  • Short fallback chains, long chains just add latency without adding reliability
  • Health checks tracking timeouts, 5xx, and 429 patterns specifically
  • Output validation before anything downstream (a tool call, a database write) touches the response
  • Logging per route: which model, latency, token usage, whether a fallback fired

Where a router fits into this

An AI gateway or model router centralizes the routing/retry/cache logic instead of every service reimplementing its own version. GonkaRouter is one example, an OpenAI- and Anthropic-compatible endpoint currently routing to MiniMax-M2.7, Kimi-K2.6, and GLM-5.2, built on the Gonka decentralized compute network. Worth being precise here: "compatible" means the request/response format matches, not that every model behaves identically to what you'd get from OpenAI or Anthropic directly, that distinction is exactly the model-context problem above, and no gateway makes it disappear on its own.

If you're evaluating something like this, the actual test is: change only the endpoint in an existing integration, run real prompts through each supported model, and measure latency, output quality, and failure behavior before you decide where your app-level retry and fallback logic needs to live.

Quick checklist before shipping failover

  • [ ] Defined behavior for timeout, 5xx, 429, and invalid JSON, not just "the request failed"
  • [ ] Retry limits with backoff, not unbounded retries
  • [ ] Fallback chains kept short
  • [ ] Context transformation handled explicitly between models, not assumed
  • [ ] Caching scoped with real invalidation rules, not just "cache everything"
  • [ ] Every route logged: model, latency, tokens, fallback fired or not
  • [ ] Output validated before it reaches a tool call or downstream write

Curious what others here are using for the context-transformation step specifically, that's the part I've seen bite people hardest when a fallback model quietly handles system prompts differently than the primary.