[Main] [Essays] [Articles] [Projects] [Now] [Resume] [Talks]

July 31 2026

The industry has a maddening relationship with solved problems. A thing gets figured out, packaged, documented, and shipped, and a decade later half the field is building it again from scratch. The old solution was rarely wrong. Somewhere along the way the proven thing got traded for something that looked cheaper, and it turned out more expensive the moment anyone counted the hours. I watch this happen on repeat and it wears on me.

Heroku did rolling deploys, blue/green releases, and health-checked rollbacks over a decade ago. You pushed to a remote and it handled the rest. Now every team running Kubernetes rebuilds that same machinery: readiness probes, deployment strategies, surge and unavailability tuning, rollback logic, and the observability to know when any of it went wrong. Kubernetes promised to hand you this. What it actually hands you is the primitives and a config surface large enough to get every piece of it subtly wrong. You traded a convenience you already had for a customization you now babysit every quarter, and somehow that counts as progress.

The Usual Suspects

Auth is the clearest case, and the one that irritates me most. There is no shortage of auth providers. Engineers write the same ad hoc routes anyway, the same users and sessions tables, the same password reset flow, the same JSON payloads, the same token refresh logic. Everyone knows the provider exists. They build it themselves regardless, ship it with at least one hole in it, and then maintain that hole forever.

Background jobs are the same story. You need to run arbitrary work outside the request cycle, so you wire together a queue, a worker pool, retry handling, dead letter behavior, and a way to see what failed. Sidekiq, Celery, and a dozen managed queues have existed for years. It gets rebuilt regardless, per stack, per team, per service.

Once you start looking, the list is long. A few others:

Retries, timeouts, backoff, and circuit breaking. Service meshes were supposed to own this so your application code never had to. Most retry logic still lives in the app, hand-written, usually without jitter and often applied to operations that were never safe to run twice.

Rate limiting. Token bucket and leaky bucket are old, well-described algorithms. Gateways offer rate limiting as a checkbox. It still gets reimplemented as middleware, often with a race condition around the counter.

Idempotency keys. Stripe wrote down the canonical pattern years ago and it is not complicated. Teams rebuild it per endpoint anyway, and get the retry-safety edge cases wrong in the process.

Feature flags. A managed service exists. Teams build a flags table and an admin page instead, then discover they have reinvented targeting rules and gradual rollout.

Caching and invalidation. Redis already exists. What gets hand-rolled is the part that was always hard: when to consider a cached value stale, and how to avoid serving the wrong thing.

Config and secrets. Parameter stores, Vault, Consul, and etcd all solve this. Every service still hand-writes its own loading, precedence, and validation, usually discovering the precedence bug in production.

Search. Postgres full text search and Elasticsearch both exist. The common path is a LIKE query, then a slow reinvention of relevance ranking once LIKE stops being enough.

State machines and workflows. Temporal and Step Functions exist to model long-running stateful work. The default is a status enum column and a growing pile of conditionals that no one can reason about after six months.

Multi-tenancy isolation. The tenant_id filter on every query is a wheel that has been built thousands of times, and the version with the security hole has been built almost as many.

The Pattern

The pattern underneath all of these is the same, and it is not subtle. A mature solution exists. The ecosystem swears a platform or provider already covers it. The engineer rebuilds it anyway, because the abstraction leaked, or did not fit the exact shape of the problem, or solved an adjacent problem and left a gap sitting right where the real work was. So you get half a solution you did not want and a second half you had to write yourself, which is the worst of both.

Keeping It Simple and Extensible

None of this is an argument for overengineering the first version. Keep it simple is still the right default, and I tell people that constantly. The point is narrower: do not engineer yourself into a box while you keep it simple. A small system and a system with every future decision already foreclosed can look identical on the first day.

Building the core product on standards and known protocols is what keeps the door open. If your auth speaks OIDC, swapping in a provider later is a configuration change. If your messaging speaks a documented protocol, adding an external system or a plugin is an integration rather than a rewrite. The trap is the opposite: hardcoded vendor dependencies, bespoke config formats, homegrown encodings, and undocumented patterns that no one wrote a test for. That version feels simple on day one and turns out to be the box, because everything you might want to add later has to be pried in against the grain.

Own It Without Rebuilding It

There is a real reason people avoid the managed version. A managed service is an external dependency that can fail on someone else's schedule, and it takes ownership of a piece of your system out of your hands. Most teams want agency over the things they run. That instinct is correct. When your auth provider has an outage, the outage is yours to explain and not yours to fix, and that is a genuinely bad position to be in.

The mistake is treating that instinct as a mandate to build from scratch. Those are not the only two options. The middle ground is a subsystem you can run locally and own end to end, without writing so much glue that you have effectively rebuilt it. Keycloak runs on your infrastructure and speaks standard protocols. Postgres, Redis, and NATS run locally and cover a large share of what the queue, cache, and messaging services were selling. You keep ownership without inheriting the maintenance burden of a bespoke implementation.

Running it yourself gets cheaper still when someone has written down how. Much of this boilerplate gets rebuilt because there is no agreed-upon document to follow, so every engineer rediscovers the same shape from first principles. An IETF-style specification, or even a well-scoped guideline, for how to build full text search or rate limiting that covers the ninety percent of use cases most systems actually have would turn a multi-week rediscovery into an afternoon of implementation. You would still write the code, but you would not be re-deriving the design every time. Stripe's idempotency-key writeup is proof it works: written down once, and anyone who reads it can build a correct version instead of a subtly broken one.

Watch the Level of Effort

The number worth watching is the ratio. Auth, queues, rate limiting, and deploys are par for the course. They are expected of any serious system and they are not the product. The effort spent on them should not exceed the effort spent on the thing that actually makes the product yours. When you find yourself several weeks deep in a hand-rolled job runner, the useful question is not whether it works but why that work is outcompeting the core product for your attention.

This is measurable, and most teams already measure it, which is the part that gets me. If your organization runs on sprint points and t-shirt sizes, the data is sitting right there. Pull the estimates and add up how much went to auth, deploy pipelines, queue plumbing, and rate limiting against the features only your product can offer. If the reinvented infrastructure is winning that comparison, your own estimates are telling you the team spent the quarter rebuilding things that already existed, and nobody read the number.

Sometimes the rebuild is justified. The proven thing genuinely does not fit, or the operational cost of adopting it exceeds the cost of a small custom version. That case is real and worth defending. The trouble is that most reinvention is not that case. It is a default reached for without checking whether the wheel already turns, and the bill comes later in maintenance, in the bug that the proven version had already fixed, and in the years of one engineer's time spent re-solving a problem that was solved before they started.

Before building the next piece of infrastructure, stop and check whether it is actually unsolved or only feels unsolved because the answer lives somewhere you did not bother to look. Most of the time it is solved. Run the proven version yourself and get back to the part of the job that is actually yours.