A quick note upfront: I'm barely scratching the surface here. I'm not a performance-engineering expert, and most of this was new to me a couple of weeks ago. But I ran into a real problem, went down a rabbit hole, and found something genuinely interesting — so I'm writing it down.

The problem: 1,000 connections, ~13 doing anything

We run a distributed web crawler. It scales out to ~100 pods, each holding ~10 database connections, so at peak we open roughly 1,000 connections to a single Postgres (Aurora) writer.

On big bursts, the database CPU pegs at 95%. The obvious reading is "we've outgrown the DB, scale it up." But when I actually looked at what the database was doing, the story flipped:

Out of ~1,000 open connections, only about 13 were ever active at the same time.

Thirteen. The other ~987 were just… sitting there. Our crawler spends almost all of its time fetching and rendering pages over the network, and only a millisecond here and there actually writing to the DB. So each connection is idle ~99% of the time. We were holding a thousand connections to do about thirteen connections' worth of work.

Connections aren't free

Here's the part I hadn't internalized: an idle connection still costs you. In Postgres each connection is a real backend process — memory, a slot in a shared array the DB scans on every query, a thing the OS scheduler has to juggle. Hold a thousand of them and the database burns a chunk of its CPU just coordinating connections, before doing any useful work.

So the 95% CPU wasn't "too much work." It was "too much overhead managing connections we weren't using." More connections were making it worse, not better. That completely inverted my mental model, which until then was basically "more concurrency = more throughput."

So why does AWS let me open a thousand connections in the first place?

This is the part that quietly misled me. If connections are costly and mostly idle, why does the database happily accept a thousand of them?

Because max_connections — the limit AWS hands you — is derived from the instance's memory, not from its ability to do work. On Aurora/RDS it's basically a formula: roughly instance memory ÷ a per-connection memory budget. A 64 GB instance can hold hundreds or thousands of backend processes before it risks running out of RAM, so that's the number you're "allowed."

That's the unlock: max_connections is a safety ceiling — "how many connections can we hold without running out of memory" — not a target. It says nothing about how much concurrent work the database can do. That second number is set by CPU, and it's far smaller — for us, ~13 active sessions, bounded by 8 cores.

They're two unrelated axes:

  • Connections are limited by memory → the number is big.
  • Useful concurrency is limited by CPU → the number is small.

Filling your connection budget doesn't mean you're using the database well. We were sitting at ~1,000 of our allowed connections and still only doing ~13 active sessions' worth of work. The generous limit was quietly inviting us to over-provision.

The analogy that stuck with me: max_connections is the number of chairs in the waiting room (limited by floor space — memory). The number of doctors (limited by CPU) is what decides how many patients actually get seen. Cramming more chairs in doesn't treat anyone faster; it just lets more people wait at once. We'd been proudly filling the waiting room and wondering why the queue wasn't moving.

So a high max_connections is permission, not a goal. The number worth chasing is active concurrency — and that's exactly what the next two ideas pin down.

Two old ideas that explained everything

I stumbled onto two pieces of theory that turned the confusion into something I could reason about. I learned most of this from these two talks, which I'd recommend:

Little's Law says the number of things in flight equals arrival rate × how long each takes (L = λ × W). Plug in our numbers — the insert rate and the ~1.5 ms each insert takes — and out pops ~13. The "why only 13 active?" mystery wasn't a mystery at all; it was arithmetic. Active concurrency is a result of throughput and service time, not of how many connections you happen to open.

The Universal Scalability Law (USL) is the one that reframed how I think about scaling. It says throughput does not keep rising as you add concurrency. Two forces fight the linear ideal:

  • contention — waiting for shared resources — which makes throughput flatten into a plateau;
  • coherency — the cost of keeping everything consistent across workers — which is quadratic, and eventually pulls throughput back down.

Written out, the whole thing is one small formula:

                    λ · N
   X(N) = ──────────────────────────────
          1 + α(N − 1) + β·N(N − 1)

Enter fullscreen mode Exit fullscreen mode

Read it as a set of dependencies. N is the only thing you control (concurrency — workers, parallel requests, active DB sessions). The other three are fixed properties of the system you're trying to discover:

  • λ (lambda) — the ideal slope. Throughput of a single worker. The numerator λ·N is the fantasy world where doubling workers doubles throughput. Everything below the line is the tax on that fantasy.
  • α (alpha) — contention. The α(N − 1) term grows linearly with N. It's the share of work that can't be parallelized — a lock, a shared resource, a serial section. As N climbs, this drags throughput toward a ceiling of ~λ/α: the plateau. (On its own, this term is Amdahl's Law.)
  • β (beta) — coherency. The β·N(N − 1) term grows quadratically (≈ βN²). It's the cost of workers keeping each other consistent — coordination, cache coherence, cross-talk. Invisible at low N; past a point it dominates the denominator and throughput actually falls. This is the retrograde — the thing Amdahl's Law misses.

How the terms trade off:

  • α = 0, β = 0 → X = λN. Perfect linear scaling. (Nobody lives here.)
  • α > 0, β = 0 → a plateau. Diminishing returns, but never worse.
  • β > 0 → the curve turns over. The peak sits at N* = √((1 − α) / β) — push concurrency past N* and you go backwards.

So α decides where it flattens, and β decides whether and where it bends back down. For us α was ~0 (no locks left — it really was all CPU) and β was small but nonzero (the coordination cost of ~1,000 backends fighting over 8 cores), which put N* right around the core count — exactly why piling on more pods stopped helping and started hurting.

The takeaway: you can model this instead of guessing

This is the bit I'm most excited about, and the reason I'm writing at all.

When you need to know "how far will this scale?", there are usually three moves:

  1. Guess / linearly extrapolate. "10 pods gave X, so 100 pods will give 10X." This is just wrong the moment you're past the knee — linear extrapolation predicts up-and-to-the-right forever, while the real curve is bending down. It's confidently, dangerously incorrect.
  2. Load test. Legitimate, but expensive and slow — spin up infrastructure, generate realistic load, run it many times, babysit it. Often you don't have the time or the environment.
  3. Model it. Take a handful of real measurements you already have, fit the USL curve, and predict the whole thing — where throughput peaks, where it goes retrograde, whether adding capacity will even help.

That third option barely existed in my head before this. You don't always need a full load-testing rig to answer a scaling question. With a few honest data points and a two-parameter model, you can get the shape: does it plateau? does it retrograde? where's the sweet spot? That's usually the exact thing you're trying to decide.

A caveat, so I don't oversell it: a model is only as good as the data you feed it, and a clean fit really wants a few well-spread measurements (a small controlled ramp beats one noisy production spike). The absolute numbers I got were rough. But even a rough fit told me the thing that mattered — this system peaks near its core count and gets worse if you push past it — which is what turned "let's just add more pods / a bigger DB" into "we have a connection-management problem, and the fix is pooling."

Wrapping up

I want to be clear that I've only scratched the surface — the two talks above (and Neil Gunther's work behind the USL) go far deeper than I can. But the shift in perspective was worth the detour: measure a little, model, predict — instead of guessing with a straight line, or brute-forcing with a load test. For a scaling decision, sometimes the cheapest useful answer is a curve you fit on the back of an envelope.