Sending a webhook is easy. You POST some JSON at a URL and move on.
Receiving one is where the bodies are buried. The endpoint is public, so anyone can call it. It gets retried, so it will run twice. It arrives out of order, so "delivered" can land before "sent". And it is on the critical path of somebody else's system, so if you are slow they will time out and retry, which makes you slower.
None of this is hard once you know it. All of it is invisible until production. Here is the complete set of things a webhook receiver has to handle, with the failure each one prevents.
The running example is email delivery webhooks (bounces and complaints), because they happen to have every awkward property at once: they are security-sensitive, they retry, they arrive out of order, and processing one twice corrupts real state. Everything here applies just as well to Stripe, GitHub, Shopify or anything else that calls you back.
1. Verify the signature, on the raw body
Your endpoint is a public URL. Without verification, anyone who learns it can post a fake "this address hard bounced" event and get a customer suppressed, or a fake "payment succeeded" and get a free subscription.
Providers sign each request with a shared secret. You recompute the signature and compare.
import { createHmac, timingSafeEqual } from "node:crypto";
export function verifySignature(
rawBody: string,
signatureHeader: string,
secret: string,
): boolean {
const expected = createHmac("sha256", secret).update(rawBody).digest("hex");
const a = Buffer.from(expected);
const b = Buffer.from(signatureHeader);
// Different lengths mean it cannot match, and timingSafeEqual throws
// rather than returning false if the lengths differ.
if (a.length !== b.length) return false;
return timingSafeEqual(a, b);
}
Enter fullscreen mode Exit fullscreen mode
Two things in there matter more than they look.
Use timingSafeEqual, not ===. A normal string comparison returns as soon as it finds a differing byte. An attacker who can measure that timing can recover a valid signature byte by byte. It is a real attack, it is easy to avoid, and the fix is one function call.
Sign the raw body, not the parsed object. This is the single most common webhook bug, and it is maddening to debug because everything looks correct.
// This breaks signature verification. JSON.parse then JSON.stringify is not
// byte-identical to what was sent: key order, whitespace and unicode escaping
// can all change.
app.use(express.json());
app.post("/webhooks/email", (req, res) => {
verifySignature(JSON.stringify(req.body), req.get("X-Signature"), SECRET);
});
Enter fullscreen mode Exit fullscreen mode
// Capture the raw bytes for this route before anything parses them.
app.post(
"/webhooks/email",
express.raw({ type: "application/json" }),
(req, res) => {
const raw = req.body.toString("utf8");
if (!verifySignature(raw, req.get("X-Signature"), SECRET)) {
return res.sendStatus(401);
}
const event = JSON.parse(raw);
// ...
},
);
Enter fullscreen mode Exit fullscreen mode
In a Next.js route handler you get this for free, because await req.text() gives you the untouched body:
export async function POST(req: Request) {
const raw = await req.text();
const signature = req.headers.get("x-signature") ?? "";
if (!verifySignature(raw, signature, process.env.WEBHOOK_SECRET!)) {
return new Response("invalid signature", { status: 401 });
}
const event = JSON.parse(raw);
// ...
}
Enter fullscreen mode Exit fullscreen mode
If your framework has body-parsing middleware enabled globally, exempt the webhook path. Every "the signature is valid in curl but fails in my app" question traces back to this.
2. Reject replays
A valid signature proves the payload came from your provider. It does not prove it is happening now. Someone who captures one request can send it again in a month, signature intact.
Providers include a timestamp in the signed payload for this. Check it:
const FIVE_MINUTES = 5 * 60 * 1000;
function isFresh(timestamp: number): boolean {
return Math.abs(Date.now() - timestamp) < FIVE_MINUTES;
}
Enter fullscreen mode Exit fullscreen mode
Use Math.abs so clock skew in either direction is handled. Five minutes is the usual tolerance: long enough to survive a slow retry, short enough that a captured request is useless by the time anyone finds it.
3. Return 200 immediately, do the work afterwards
Providers give you a short timeout, often 5 to 30 seconds. Go over it and they record a failure and retry. If your handler is slow because it sends an email, updates three tables and calls another API, you will get retried while the first attempt is still running, and now you have two of them.
Acknowledge first, process after:
export async function POST(req: Request) {
const raw = await req.text();
if (!verifySignature(raw, req.headers.get("x-signature") ?? "", SECRET)) {
return new Response("invalid signature", { status: 401 });
}
const event = JSON.parse(raw);
// Durable queue, not a floating promise. If the process dies between the
// 200 and the work, a background job survives it; a stray async call does not.
await enqueue("webhook-events", event);
return new Response("ok", { status: 200 });
}
Enter fullscreen mode Exit fullscreen mode
The enqueue has to be durable. processEvent(event) without an await, fired off before returning, will silently lose events on deploy, restart or crash, and you will never know which ones.
Return the right status. 200 means "I have it, stop retrying". A 4xx means "this is broken, do not bother retrying". A 5xx means "try me again". Returning 200 on an error you could have recovered from throws the event away permanently.
4. Assume every event arrives twice
Retries are not an edge case. They happen on timeouts, on deploys, on network blips, and some providers retry on a schedule for hours. Your handler will run twice on the same event, and it must not do the work twice.
Do not solve this with a "have I seen this?" check in application code. Two concurrent retries will both read "no" before either writes. Let the database decide:
CREATE TABLE processed_webhook_events (
event_id TEXT PRIMARY KEY,
processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
Enter fullscreen mode Exit fullscreen mode
async function handleOnce(event: WebhookEvent) {
const claimed = await db
.insertInto("processed_webhook_events")
.values({ event_id: event.id })
.onConflict((oc) => oc.doNothing())
.executeTakeFirst();
// Somebody else already claimed this id: a retry, or a concurrent delivery.
if (claimed.numInsertedOrUpdatedRows === 0n) return;
await doTheActualWork(event);
}
Enter fullscreen mode Exit fullscreen mode
The primary key does the work. Whichever request inserts first wins, the other returns immediately, and no amount of concurrency changes that.
If the provider does not send a stable event id, build one from fields that identify the occurrence: the message id plus the event type plus the timestamp. Do not hash the whole payload, because providers add fields over time and your fingerprint would change for what is really the same event.
5. Events arrive out of order
There is no ordering guarantee. A retried sent from three minutes ago can land after the delivered that followed it. Naive handling walks the status backwards:
// Wrong: last writer wins, whatever it says.
await db.update(emails).set({ status: event.type }).where(eq(emails.id, id));
Enter fullscreen mode Exit fullscreen mode
An email that was delivered, then bounced, then had its sent retried will finish as "sent". Every dashboard is now wrong.
Fix it with precedence, not timestamps. Some states are terminal and outrank late arrivals:
const RANK = { queued: 0, sent: 1, delivered: 2, complained: 3, bounced: 4 };
function nextStatus(current: Status, incoming: Status): Status {
return RANK[incoming] > RANK[current] ? incoming : current;
}
Enter fullscreen mode Exit fullscreen mode
Now a late sent cannot overwrite bounced, and the order events happen to arrive in stops mattering. Provider timestamps work too, but they come from someone else's clock, and precedence encodes what you actually mean.
6. Fail loudly, and keep what you could not process
Events you reject are gone. The provider retries a few times and gives up, and by then nobody is looking. Store what you could not handle:
try {
await handleOnce(event);
} catch (err) {
await db.insert(failedWebhookEvents).values({
eventId: event.id,
payload: JSON.stringify(event),
error: String(err),
});
throw; // 5xx so the provider retries as well
}
Enter fullscreen mode Exit fullscreen mode
Two reasons this earns its place. You can replay after fixing the bug, and a sudden pile of rows in that table is the clearest possible alert that something changed on their side.
Testing it locally
You cannot receive a webhook on localhost from the internet, so use a tunnel:
# cloudflared, ngrok or similar: whatever puts a public URL on your local port
cloudflared tunnel --url http://localhost:3000
Enter fullscreen mode Exit fullscreen mode
Register the printed URL as your webhook endpoint in the provider's dashboard and trigger a real event.
For the tests that matter, skip the network entirely. Signature verification, idempotency and ordering are pure functions of the payload, so test them directly:
it("ignores a replayed event", async () => {
const event = { id: "evt_1", type: "bounced", messageId: "m1" };
await handleOnce(event);
await handleOnce(event); // the retry
expect(await countSuppressions("m1")).toBe(1);
});
it("does not let a late sent overwrite a bounce", () => {
expect(nextStatus("bounced", "sent")).toBe("bounced");
});
Enter fullscreen mode Exit fullscreen mode
These run in milliseconds, need no tunnel, and cover the failures that actually happen in production.
The checklist
- Verify the signature on the raw body, with a constant-time compare.
- Reject events outside a few minutes of now.
- Return 200 fast, do the work in a durable queue.
- Deduplicate on the provider's event id, enforced by a unique constraint.
- Rank your states so late events cannot walk them backwards.
- Persist failures so you can replay them and notice them.
Most providers document their half of this well. SMTPfast's delivery webhooks send a signed payload with a stable event id per delivery, bounce and complaint, which is what makes points 1 and 4 straightforward, and it is worth checking your provider gives you both before you build against them. If they do not send a stable event id, you have to synthesise one, and that is a good thing to find out on day one rather than after your first duplicate.
The pattern is the same everywhere: an endpoint anyone can call, that runs more than once, in an order you do not control. Build for that from the start and webhooks are boring. Discover it in production and they are the thing that quietly corrupted a week of data.
0 Comments
Log in to join the conversation.No comments yet. Be the first to share your thoughts.