How Spotify Deployed Kong's AI Gateway to Power Generative AI at Scale

The Challenge: AI Adoption Across Spotify 

Spotify's goal for AI is bold, but simple: make it as intuitive and indispensable to daily work as email or Google Docs. And not just for engineers, but for every Spotifier across every function.

That vision is driving the AI-transformation strategy, company-wide. The initiative is investing in AI education for 7,000+ Spotifiers, rolling out agentic IDEs to 100% of developers, building shared AI-powered tools for business operations, and establishing the secure foundations for an AI-native workplace. This goes beyond a tooling rollout or a training campaign: it's a fundamental shift in how the company operates.

But ambition at this scale creates an infrastructure problem. Gen AI-powered features are increasingly integrated across various parts of Spotify, including external areas like music, podcasts, ads, and search, as well as internal functions such as legal, social, marketing, and finance tools. 

Behind each of those features, there's a team that needs to talk to an LLM. A year ago, that meant each team was on its own. One squad would integrate directly with OpenAI's API, another with Vertex AI, another with our first-party Hendrix models. Every team resolved the same problems: how to authenticate, how to handle rate limiting, how to track costs, how to fail over when a provider goes down. There was no centralized observability, no consistent error handling, no uniform safety policy, and large governance overhead as well.

It worked when AI usage was a handful of experiments. It stopped working as hundreds of consumers began onboarding with fragmented LLM inference requirements — from AI-powered search to ad generation to DJ narration to internal developer tools. 

Every new integration meant weeks of boilerplate. Every outage at a model provider triggered a scramble across unrelated teams. And when leadership asked how much we were spending on LLMs, nobody had a single answer.

Without shared infrastructure, Spotify risked exactly what the strategy was created to prevent: fragmented adoption, duplicative tool development, and operational inefficiencies. We needed a gateway.

Why Kong: Building on Proven API Infrastructure 

One of the five strategic pillars for the AI-transformation strategy is "Building the AI Engine" — creating the shared, secure foundations that allow teams across the company to move fast and safely with AI. A major component of that engine is the AI Gateway, which launched in May 2025. When we evaluated how to build it, we had a clear build-vs-buy decision to make. We could write a custom proxy from scratch, or we could stand on the shoulders of something battle-tested.

Performance benchmarks confirmed the choice. Kong demonstrated ~40ms routing overhead and was expected to scale beyond 1,000 RPS per instance — more than enough for our traffic patterns. And because Kong is fully self-hosted, all data stays within Spotify's infrastructure — a non-negotiable requirement for a system handling every LLM prompt in the company.

We started with Kong OSS with a clear path to enterprise features if needed. As we scaled, we realized that there were features we required that weren't available open source. Because we were running business-critical traffic through the AI Gateway, we needed enterprise-grade security and support. That was when we talked with the Kong team and made the move to the enterprise version. Here's what made it the right fit:

• Hybrid mode fits our deployment model. We run on GKE, and Kong's separation of control plane (configuration management, backed by PostgreSQL) and data plane (stateless proxies handling traffic) mapped directly to how we wanted to deploy. The control plane manages config and connects to CloudSQL. The data planes scale independently and handle all client traffic. No shared state between proxy pods means we can scale horizontally without coordination.

• Declarative configuration fits our GitOps workflow. Kong's decK tool lets us define every consumer, route, service, and plugin in version-controlled YAML. Every change goes through a pull request, gets validated automatically, and rolls out through our CI/CD pipeline. No one clicks buttons in a UI to change production routing.

• The plugin ecosystem gave us extensibility without forking. Kong ships with a rich set of bundled plugins, and we can build custom ones in Lua. We needed authentication, guardrails, and provider-specific logic — the plugin model let us compose these cleanly instead of building a monolith. We have currently developed plugins that provide security and safety capabilities.

• AI-native capabilities accelerated our timeline. Kong's ai-proxy plugin handles multi-provider routing and request transformation. The Prometheus plugin's ai_metrics: true flag gives us token-level telemetry out of the box. We didn't have to build AI-specific observability from scratch.

• A standard interface for model switching. One of the deciding factors was Kong's ability to provide a single API format while transparently routing to different providers — first-party Hendrix models, OpenAI, or Vertex AI. Teams integrate once and can switch providers without code changes. This was critical for the early use cases like AiKA (Spotify's internal AI knowledge assistant) and our Prism models that we needed to onboard first.

Architecture: A Gateway Purpose-Built for AI Traffic

AI Traffic is Not Web Traffic

Before diving into the architecture, it's worth explaining why a general-purpose API gateway isn't enough for AI workloads. AI traffic has fundamentally different characteristics:

These differences drove nearly every tuning decision we made.

The Architecture

The Plugin Chain: Infrastructure for Unified AI Governance

Every request through the gateway passes through a chain of plugins that enforce authentication, authorization, safety, and observability. Here's an example of what it looks like in practice:

1. ai-proxy-advanced (Multi-Provider Routing) The heart of the AI Gateway. A single API format on the client side, with provider-specific authentication and request/response transformation on the backend. Teams write to one API, and we route to Hendrix, OpenAI, or Vertex AI transparently.

2. prometheus (AI Metrics) Standard HTTP metrics plus AI-specific telemetry: total LLM requests per model (kong_ai_llm_requests_total), token consumption by type (kong_ai_llm_tokens_total), provider latency histograms (kong_ai_llm_provider_latency_ms). Per-consumer metrics are enabled, so we can attribute usage down to individual service accounts.

3. Open telemetry (Distributed Tracing & LLM Observability) End-to-end tracing from client through the gateway to the model provider. We sample at 1% to keep costs manageable while still having enough traces for debugging.

Impact: What Changed

The AI gateway is the shared foundation that the initiative's strategic pillars depend on. When we talk about "supercharging product innovation" and "reimagining business operations" with AI, this is the infrastructure that makes it possible.

Over 1,000 internal services and individuals have been onboarded, and that number continues to increase — from AI-powered search to ad creative generation to podcast transcription to internal developer tools to security scanners. The diversity reflects the breadth of the initiative's ambition: DJ narration, playlist DNA analysis, SOX compliance copilots, UX copy generation, flaky test analysis, and HR chatbots.

Time saved on build-vs-buy alone. And that's just the gateway itself. For each consuming team, what used to take weeks of provider-specific authentication, error handling, and retry logic is now a config change: add a consumer to consumers.yaml, open a PR, merge. A team can go from zero to production in an afternoon.

Consistent security and safety are applied uniformly. The ai-guardrails plugin scans every prompt before it reaches any model. No team can accidentally bypass content safety checks because the checks are in the infrastructure, not the application code.

Model provider flexibility. Teams experiment with different providers without changing their integration. When we onboard a new model or provider, every consumer gets access through the same API. When a provider has an outage, we can reroute at the gateway level rather than asking 800+ consumers to update their code.

"There's a lot at stake when you are running a generative AI program at the scale of Spotify's. By partnering with Kong and deploying their AI gateway, we democratize access to APIs and easy governance at scale. We've saved hours of engineering time, created consistent security protocols, and ensured we have the model flexibility we need to move and innovate fast."

Mike Seid, Product Area Tech Lead, Machine Learning Platform at Spotify

What's Next

The gateway is stable and growing, but we're not done. The strategy was focused on moving Spotify from broad AI adoption to deep, high-impact usage — and the gateway's role is expanding to match.

More providers and capabilities. We continue to expand the set of supported model providers and model types. As new modalities emerge — image generation, audio, video — the gateway will be the integration point.

Deeper confidence integration. We want to connect the gateway to Spotify's experimentation platform, Confidence, for model A/B testing. Route a percentage of traffic to a new model, measure quality and latency, promote or rollback — all controlled at the gateway layer.

Fortifying the agentic foundations. The initiative's vision for the next phase includes empowering Spotifiers across all technical abilities to build, release, and maintain AI agents that automate their workflows. As these agents become first-class API consumers — making chains of calls, using tools, maintaining state across interactions — the gateway's role in authentication, rate limiting, and observability becomes even more critical. An agent that can make unbounded API calls needs guardrails that the application alone can't enforce. The gateway is already part of the trusted agentic environment we’re building: strong access controls, observability for AI-driven risks, and a golden path for secure AI usage.

Conclusion

Building this gateway wasn't just about managing traffic; it was about shifting our engineering culture to treat AI infrastructure with the same rigor as our traditional web services. By centralizing authentication, observability, and configuration, we’ve removed the friction that previously slowed down our teams. As we move toward more complex agentic workflows, this foundation provides the security and flexibility we need to scale safely. We’ve successfully turned AI from a series of fragmented experiments into a core, scalable engine that empowers every team at Spotify.

--

This post reflects the work and thinking of the entire AI Gateway team. Thanks to Aamir Anwar, Maria Gordiyenko, Taylor Page Hendricks, Johnathan Howard, Kapil Mirchandani, Ikenna Odiakosa, Lucas Rea, and Guy Stone for their contributions.