Almost every search application is multi-tenant, whether or not anyone called it that. A store has product categories. A SaaS product has customers. A logging platform has teams, an e-commerce site has sellers, a B2B app has accounts. Each of those is a tenant: a slice of data, and the queries that go with it, that has to stay separate from every other slice. If you run Amazon OpenSearch Service, odds are you are already in the tenancy business.

And here is the thing about tenancy: it is almost free when you have a few tenants, and it quietly turns into the whole problem when you have a lot of them. A design that is clean and obvious at a hundred tenants can fall apart at a hundred thousand, for reasons that have nothing to do with the code being wrong. Take a concrete case. You are running a fintech app, peer-to-peer payments or a neobank, and millions of users each expect to open their phone, pull up six months of history, and see it instantly. That data behaves like logs, streaming in and aging out, but it has to perform like search, because a person is staring at a spinner. It lives in the gap between the two, and the conventional wisdom for either end does not fit.

I have watched teams wrestle with this exact problem. They stand up a textbook Amazon OpenSearch Service domain, follow every log-analytics best practice, demo it, and it screams. Beautiful on the happy path, right up until a certain scale. Then the queries slow down, the costs balloon, and the architecture that purred along at 10,000 tenants starts making the noise hardware makes right before the magic smoke escapes. The question I hear next is always the same: “We thought we were doing this right. What changed?” Usually nothing changed. The design followed the playbook faithfully, right down to copying the shard sizes from a blog post. The playbook just was not written for this workload, and scale is where that bill comes due. Let me walk through where the conventional wisdom runs out, and then the handful of moves that actually hold up at a million tenants.

Where the conventional wisdom runs out

The first trap is the pure-logging approach. The standard playbook says ingest fast, roll over indices daily, keep three days hot, then tier the rest off to cheaper storage. Shard size? Go big, 50 GB is fine, because you are optimizing for write throughput. This is excellent advice for application logs that nobody reads unless something is on fire.

It falls apart the moment you serve individual tenant queries at scale. Every query fans out across every shard in the index. A terabyte in 50 shards means one user tapping their phone becomes a single request fanning out to 50 servers, all to answer somebody checking last month’s coffee purchases. The cost of coordinating that scatter-gather swamps the benefit of parallelism, especially when each tenant’s data is a thin slice smeared across all 50 shards. The cluster was tuned for throughput, and now it is being asked for the one thing that tuning works against.

The second trap is the mirror image: the pure-search approach, giving every tenant its own index. Perfect isolation, fast queries, and it works right up until a few 10s of 1000s of tenants, at which point the cluster falls over under the weight of managing so many indices. OpenSearch Service tracks metadata for every index, shard, and replica, and the cluster state balloons until it becomes the bottleneck. Now cross-tenant queries are a coordination nightmare and the master node spends its life just keeping track of everything.

The third trap is the subtlest, and the easiest to walk into: assuming a tiering strategy built for logs will hold up for user-facing queries. Teams design a gorgeous multi-tier system: hot for recent data, UltraWarm for older data, maybe cold storage for compliance. Then they discover UltraWarm adds seconds of latency, because it pulls from S3 on demand. Seconds are fine for a forensic investigation. Seconds are a catastrophe when a user is staring at a spinner waiting for last spring’s transactions.

The conventional wisdom says pick your poison, optimize for cost or optimize for latency. At a million tenants you need both, which is exactly why the standard playbooks leave you stranded by the side of the road.

The right way to think about tenant distribution

The unlock is admitting that not all tenants are created equal, and your architecture should say so out loud. The move that changes everything is routing, in the broad sense: dedicate a slice of resources to match each slice of the workload, then get every request to the right slice. Everything below is a way to do that. Custom document routing does it inside one domain. Multi-domain and cell architectures do it across many. Pick the tool that fits the shape of your tenants.

Start with the within-a-domain tool, custom document routing. It is the right lever when you are pooling many tenants into shared indices, the case where tenants vastly outnumber indices. By default, OpenSearch Service spreads documents across shards with a hash function. Great for even distribution, terrible for tenant isolation. Custom document routing lets you supply your own hash key, usually a tenant ID, so all of a tenant’s data lands on one shard. When that tenant queries, the request hits one shard instead of all 50. One query, one shard, instead of fifty servers woken up to answer it. One shard is not the only option, either. Set index.routing_partition_size and a routing value maps to a subset of shards rather than a single one, spreading a larger tenant across, say, three shards instead of one. You give up a little of the single-shard win, since queries now fan out to the subset, in exchange for spreading a big tenant’s data and load. It is the same routing idea with a dial on it: size the slice of shards to the size of the tenant.

Do the arithmetic, because the arithmetic is the whole argument. A million small tenants on random distribution means every query touches every shard: a million queries, each firing 50 network calls. With routing, each query hits one shard. You cut the networking cost by around 98%, and latency drops with it. That is not a tuning tweak. It is the difference between a system that works and one that does not, and it is one of those rare architecture changes where the math is so lopsided it feels like cheating.

The nuance is that routing only earns its keep past a certain scale. The tipping point usually sits in the tens of thousands of tenants. Below that, fussing over routing keys is effort you will not get back. Above that, it is survival. I have watched this pattern work beautifully for financial-services companies with millions of end users, where each user’s data is small but has to come back now.

Pooling handles the small tenants, but the big ones need the opposite treatment, and that is where the hybrid model comes in. The idea is to dedicate a slice of resources to each large tenant independently of the small ones, so a whale’s spikes and volume never land on the shards the minnows share. Give that tenant its own dedicated index, sized and replicated for its workload, and its load is isolated from everyone else’s. Pool the long tail of small tenants into shared indices with custom routing. Silo the whales, pool the minnows, and match the size of the resource slice to the size of the tenant. A dedicated index is the within-a-domain way to hand a tenant its own slice. When a tenant, or a group of them, needs more isolation than one index inside a shared domain can give, the slice grows to a whole domain of its own.

Hybrid is a spectrum: routing within one domain at one end, a cell per group of tenants across many managed domains in the middle, and a collection per tenant on serverless at the other. That middle option is a cell architecture: each domain is a cell that owns one group of tenants, and a routing layer in front sends each request to the cell that holds its data. The Amazon Your Orders team built exactly that for semantic order-history search across billions of records, in improving order history search with semantic search. At the serverless end, Amazon OpenSearch Serverless can run a collection per tenant behind one regional endpoint, so routing is a request header and scale-to-zero means an idle tenant costs only storage, which I cover in multi-tenant search with OpenSearch Serverless next generation. Same logic throughout: silo the whales, pool the minnows, and pick the container that fits each tenant.

Mixed workloads create real tension in shard sizing, because you are optimizing for write throughput and read latency at the same time. Write-heavy wants more, smaller shards to parallelize ingestion. Read-heavy wants fewer, larger shards to keep queries from fanning out. When both happen at once, you land in the middle on purpose. My heuristic: aim for 30 to 40 GB per shard, rather than the 50 GB you would use for pure logs or the 10 to 30 GB you would use for pure search. Benchmark against your own workload, because your mileage will absolutely vary, but that middle band tends to balance ingestion against query performance. You are not optimizing for either extreme. You are optimizing for reality, where both matter.

Tiering has a better answer now, too. Writable warm storage runs on the OpenSearch Optimized OI2 instance family and is not the old UltraWarm. The big difference is not speed. Both tiers are S3-backed with a local cache, and on the NYC Taxis benchmark writable warm matches or beats UltraWarm on most query types, so call it comparable, not faster. The difference that matters is mutability. UltraWarm is read-only, so touching a single old document means migrating the index back to hot, writing, and migrating back, a warm-to-hot-to-warm round trip that can take a couple of hours per 100 GB. Writable warm lets you write directly to warm. Late-arriving data, compliance corrections, and backfills resolve in seconds instead of hours, with no migration and no hot-node cost. For a fintech workload where a six-month-old transaction occasionally has to be corrected, that is the whole ballgame: you keep the S3 cost profile and lose the read-only handcuffs.

Where this is heading

The patterns for multi-tenant log analytics are still evolving. More teams are converging on this hybrid shape: pooled tenants with routing, dedicated resources for the power users, and tiering driven by real access patterns instead of age alone.

The next frontier is dynamic routing that reacts to tenant behavior. Picture a system that promotes a tenant from pooled to dedicated the moment its query volume spikes, then quietly demotes it when things calm down, with no operator intervention. Or routing that respects geography and time zones, noticing that your Asia-Pacific users are asleep while your North American users are hammering the cluster.

If you are building one of these systems now, instrument everything. You need per-tenant visibility into query latency, shard distribution, and resource use, because the difference between an architecture that works and one that does not is usually hiding in the metrics nobody thought to collect. Start with the hybrid model. Turn on custom routing once you cross five-digit tenant counts. And do not assume a tiering strategy designed for logs will survive contact with a user-facing query.

The million-tenant problem is not unsolvable. It just refuses to be solved by the idealized architecture in the docs, the one that works great in the diagram and has never met real traffic. Build for the workload you actually have, not the one in the tutorial.