Hardil Singh

Working on systems that move real money changes what "saved to the database" is allowed to mean. A row existing in Postgres is not the same claim as "this record is durable and this record is true." Here's the pattern I now apply to any code where a mutation has to be trusted after the fact — balances, ledgers, anything that gets audited.

One choke point, always

Every mutation of critical state — a credit, a debit, a transfer, a reversal — goes through exactly one audited function. Not "usually," not "except for this one admin shortcut." One function, and every caller, including internal tooling, goes through it. The moment a second code path can also mutate the balance, the ledger stops being a source of truth and becomes one of several competing opinions about what happened.

Caches are allowed to exist for read performance. They are never allowed to diverge from the ledger, and they're never the thing you trust when reconstructing history.

Append-only and hash-chained, not just "logged"

A row that can be edited or deleted after the fact isn't a record, it's a draft. Each ledger entry includes a hash of the previous entry, so any modification to history — not just deletion, but an in-place edit of an old row — breaks the chain and is immediately detectable. Where the integrity guarantee needs to be verifiable by someone outside the system (a counterparty, an auditor, a regulator), the chain gets signed, verifiable with a public key and no shared secret required.

Durable means it survives a crash between "accepted" and "written"

A record isn't durable the moment your code decides to write it — it's durable once it's actually persisted somewhere that survives a crash. The gap between those two moments is where money quietly disappears in systems that don't think about it. The pattern: a durable enqueue with a local write-ahead-log fallback, so if the primary write path fails partway, nothing is silently dropped — it gets recovered and replayed on the next boot.

Nothing that's genuinely "of record" is allowed to live only in memory. If a process restart can make a transaction vanish, it was never actually recorded — it was a promise.

Sequence gaps are fine to detect; collisions are not fine to allow

Sequence or ID design should make gaps detectable — a monotonic counter where a skipped number is a visible anomaly worth investigating — while making collisions structurally impossible, not just statistically unlikely. A detectable gap is an incident to review. A silent collision is corrupted history.

Decoupling the hot path from the database

Under load, the request/response path should never be gated on a synchronous database write. High-frequency or non-critical events go through a durable, backpressured queue with a single batched writer — bursts queue up, the database sees smooth load instead of a spike, and nothing on the user-facing path stalls waiting on persistence. The queue is still durable end to end; "decoupled" doesn't mean "best-effort."

Fairness needs the same rigor as money

Anywhere an outcome has to be provably fair (a random draw, a game result, anything adjudicated after the fact), the seed has to come from a real CSPRNG, generated per-event — never a fixed value, and never derived from something observable or predictable like the system clock. Log the seed and the ruleset in force at the time, so a disputed outcome can actually be reconstructed and verified later, not just asserted.

What I'd tell past-me

  • "It's in Postgres" is not the same claim as "it's durable." Ask what happens if the process dies one line after the write call.
  • A single audited choke point isn't bureaucracy — it's the only way "reconstruct what happened from the ledger alone" stays true as the codebase grows.
  • Build the tamper-evidence in from the start. Retrofitting a hash chain onto years of mutable history is a much harder project than doing it on day one.