Max Bob

If you build anything on top of hosted AI APIs, latency isn't a detail you get to ignore — it's a feature. A sluggish time-to-first-token is the difference between an assistant that feels alive and one that feels broken.

Yet when I went looking for an honest answer to a simple question — how fast is provider X from where my users actually are? — I couldn't find one. The numbers people quote tend to come from a single machine in a single region (usually somewhere in the US), from vendor-reported status pages, or from a benchmark that got run once and never refreshed. There was no independent, measured, regional view.

So I built one: LLM Latency Tracker, a provider-neutral tracker of latency and uptime for AI inference APIs.

How it works

The core idea is boring on purpose: actually measure, don't scrape.

A small Python prober — standard library only, no API key needed for the edge probes — opens real connections to each provider's endpoint and times every phase of the handshake: DNS resolution → TCP connect → TLS negotiation → time-to-first-byte (TTFB). That's the edge view: how long the network path itself takes before a single byte comes back.

Separately, where possible, it measures inference time-to-first-token (TTFT) — the thing your users actually feel, i.e. how long after you hit "send" the model starts streaming.

Those two numbers answer different questions, and keeping them apart matters. Edge latency is about the network and the front door; TTFT is about the model and the queue behind it.

The probes run from four regions — Europe (Germany), US Central, Asia (Tokyo), and South America (São Paulo) — because "fast" is meaningless without "from where." Results land in a SQLite time-series; a static-site generator turns that into the pages you see, hosted on Cloudflare Pages, and the whole thing is self-updating on a schedule. There's no always-on backend to rot or page me at 3am.

It currently covers ~45 providers — the usual Western labs (OpenAI, Anthropic, Google, Mistral, Groq, Together, Fireworks, Cerebras, OpenRouter, Perplexity, xAI, DeepSeek…) plus Chinese ones (GLM/Zhipu, Kimi/Moonshot, Qwen, MiniMax) that rarely show up in Western benchmarks. All of it is free and open under CC-BY-4.0, with a JSON API and an OpenAPI spec.

The interesting part: building it agent-native

Here's what makes this more than another dashboard. I built it to be read by machines, not just humans.

The way people ask questions is shifting from search boxes to answer engines and agents. If your data only exists as HTML wrapped in JavaScript, an agent can't reliably consume or cite it — which means your measurements never reach the place where someone actually asks "which provider has the lowest TTFT in Europe right now?"

So the site speaks several agent dialects:

  • A real MCP server (Streamable HTTP) at /mcp, exposing a get_ai_api_latency tool. An agent can call it directly and get structured latency data back — no scraping, no guessing.
  • An MCP Server Card at /.well-known/mcp/server-card.json, plus WebMCP, so agents can discover the tool.
  • llms.txt and llms-full.txt, giving models a clean map of the site.
  • Markdown content negotiation: ask a page for text/markdown and you get tidy Markdown instead of a DOM you have to reverse-engineer.

The nice side effect is that building for agents forces you to build clean for everyone. The same discipline that makes a page legible to an LLM — structured, fast, no junk — is what earned a Lighthouse 100 and Cloudflare's "Agent-Native" (Level 5) rating on their "Is Your Site Agent-Ready?" check. I won't pretend that means much on its own; it's a checklist, not a trophy. But it's a decent proxy for "this data is actually reachable."

There's also a small companion: a model deprecation & migration calendar at /deprecations, hand-verified from official provider docs, so you can spot which models are being sunset before they break your app.

Honest limitations

A tracker that oversells itself is worse than none, so — plainly:

  • The dataset is young. It's only days old as I write this. Treat trends as provisional; don't over-read a single snapshot.
  • Edge latency is not a full round-trip. TTFB to an endpoint measures the network path, not the time to a finished generation. TTFT is closer to lived experience, but still isn't total completion time.
  • Keyless network probes and keyed inference probes see different things. Inference coverage is narrower than edge coverage.
  • Four regions is a sample, not the planet. If your users sit somewhere I don't probe, your mileage will differ.

It's a baseline and a starting point, not gospel.

Try it / break it

It's free, open-source, and the data is CC-BY-4.0 — use it, chart it, cite it, build on it.

If a provider or region you care about is missing, open an issue or a PR — that's exactly the kind of contribution that makes this more useful for everyone. And if you're building agents, point one at the MCP endpoint and tell me what breaks.