Make an Outbox Relay Lease Expire Without Violating Delivery Invariants

Relay A claims outbox row 42 and publishes its message. Before A records completion, its lease expires. Relay B claims row 42 and publishes it again. Both relays behaved according to their local state; the broker received a duplicate.

A lease prevents unlimited concurrent ownership. It cannot create exactly-once delivery across a database and broker without a shared transaction.

The honest contract is:

Every committed outbox event is eventually offered at least once; consumers preserve business invariants under duplicate offers.

State model

pending
  -> leased(owner, token, expires_at)
  -> published

leased --expiry--> claimable
leased --failure--> claimable

Enter fullscreen mode Exit fullscreen mode

A fencing token increments on every claim:

UPDATE outbox
SET lease_owner = :worker,
    lease_token = lease_token + 1,
    lease_expires_at = now() + interval '30 seconds'
WHERE id = :id
  AND published_at IS NULL
  AND (lease_expires_at IS NULL OR lease_expires_at < now())
RETURNING id, event_id, payload, lease_token;

Enter fullscreen mode Exit fullscreen mode

Completion is conditional:

UPDATE outbox
SET published_at = now()
WHERE id = :id
  AND lease_owner = :worker
  AND lease_token = :token
  AND published_at IS NULL;

Enter fullscreen mode Exit fullscreen mode

The token stops stale relay A from marking B's newer lease complete. It does not retract A's already published message.

Minimal event-order simulator

Model actions instead of threads:

state = {
    "token": 0,
    "lease": None,
    "published": False,
    "broker": [],
}


def claim(worker):
    state["token"] += 1
    state["lease"] = (worker, state["token"])
    return state["token"]


def send(worker, token):
    state["broker"].append({"event": "e42", "worker": worker, "token": token})


def complete(worker, token):
    if state["lease"] == (worker, token):
        state["published"] = True


a = claim("A")
send("A", a)
# Time advances; A's lease expires.
b = claim("B")
send("B", b)
complete("A", a)  # fenced out
complete("B", b)

assert len(state["broker"]) == 2
assert state["published"] is True

Enter fullscreen mode Exit fullscreen mode

The duplicate is not a simulator bug. It is the counterexample that defines the consumer requirement.

Consumer invariant

Suppose the event awards a $10 credit. The wrong handler performs an unconditional increment twice. A robust handler stores event identity in the same transaction as the business effect:

BEGIN;

INSERT INTO consumed_events (consumer, event_id)
VALUES ('credit-service', 'e42')
ON CONFLICT DO NOTHING;

-- Continue only when the insert affected one row.
UPDATE accounts SET credit_cents = credit_cents + 1000 WHERE id = :account;

COMMIT;

Enter fullscreen mode Exit fullscreen mode

The invariant is not “the handler ran once.” It is:

For each (consumer, event_id), the credit effect commits at most once.

Enter fullscreen mode Exit fullscreen mode

Other event types need different invariants. A state projection can use monotonic versions. A notification may need a delivery ledger. A cache invalidation can be naturally idempotent. Do not apply one generic deduplication recipe without naming the business effect.

Failure classes to inject

Event order	Expected property
send, crash, lease expiry, resend	one business effect
claim A, expire, claim B, A completes	stale completion rejected
broker acknowledgement lost	resend allowed
consumer commits effect, acknowledgement lost	redelivery causes no second effect
duplicate arrives concurrently	uniqueness constraint serializes effect
event arrives after newer version	projection does not move backward

Generate permutations around claim, send, complete, consume, and acknowledge. The acceptance rule should inspect durable state, not log-line counts.

Tradeoffs

Mechanism	Helps	Does not solve
relay lease	bounded ownership, crash recovery	post-send ambiguity
fencing token	stale database completion	duplicate broker message
broker deduplication	duplicates within broker window	long delays or consumer effects
consumer inbox	effect deduplication	unbounded retention
monotonic version	stale projection updates	additive non-idempotent effects

Inbox retention must cover the maximum plausible redelivery interval. Deleting deduplication records after one day while a dead-letter queue retains messages for a week reopens old effects.

Limits

This model assumes the source transaction atomically writes domain state and the outbox row. It does not address event ordering across aggregates, broker partition changes, poison messages, or multi-region conflict resolution. Those require additional invariants.

Exactly-once can sometimes be provided inside one technology boundary. Across independently failing database, relay, broker, and consumer boundaries, state the narrower guarantee and test the violating event order. For this protocol, lease expiry after send is the sequence every design must survive.