Skip to content
Jalel Dridi
All writing

· 4 minute read

Webhooks arrive in any order. The end state should not care.

Two kinds of out-of-order delivery in a payments system, and the two rules Payout Ledger uses to end in the same place every time.

  • Webhooks
  • Stripe
  • State machines

Stripe does not guarantee the order in which it delivers events. A payout.paid can arrive before the payout.created it follows, and a delayed payout.updated can arrive after payout.failed. Apply events as they come and a stale one overwrites a newer state.

In Payout Ledger, my open-source payout monitor, I treated this as two separate problems, because it is: disorder inside one object, and disorder between objects.

Inside one payout: only move forward

  1. Order by the event's timestamp. Ignore any event older than the last one applied. But the timestamps have one-second resolution, so two events in the same second cannot be ordered.

  2. Refetch the object from Stripe. Treat the event as a signal and read the current state from the API. Always correct for real objects, but it costs an API call per event and cannot work for simulated events, which have no object behind them.

  3. A one-way state machine. An event may only move a payout forward; anything else is recorded and skipped. The cost: the allowed transitions are defined and kept correct by hand.

I chose the state machine. The whole of it fits on a screen:

src/payouts/transitions.tsTypeScript
const NEXT: Record<PayoutStatus, readonly PayoutStatus[]> = {
  pending: ["in_transit", "paid", "failed", "canceled"],
  in_transit: ["paid", "failed", "canceled"],
  paid: ["failed"],
  failed: [],
  canceled: [],
};

export function canTransition(from: PayoutStatus, to: PayoutStatus): boolean {
  return NEXT[from].includes(to);
}

An event for a payout that does not exist yet creates it in the state the event describes, so a late payout.created is a no-op, not an error. Skipped events stay in the inbox, so nothing is lost.

There is one exception to "forward only", and it is Stripe's own: a paid payout can later fail when the bank returns it. That transition is allowed explicitly, and it reverses the ledger entries.

Between objects: fail, then retry

The second kind of disorder crosses objects. A payout event can arrive before the transfer that funded the seller's balance. Applying it at once would drive the balance negative, which the ledger forbids.

  1. Buffer and reorder. Hold events and release them in dependency order. You need to know the dependency graph up front, and how long to wait for an event that may never come.

  2. Allow temporary negative balances. Apply everything on arrival and expect it to net out. That gives up the never-negative rule, so a real overdraft can no longer be told apart from a timing artefact.

  3. Fail and retry. Applying the event fails with "insufficient funds" and rolls back completely. The event stays in the inbox with an attempt count and the error, and a later sweep retries it, oldest first.

I chose fail and retry. Each event is applied in a single transaction, so a failed attempt leaves no partial state. After eight failed attempts the sweep stops and leaves the event for a person to inspect.

The webhook still answers 200. The event was received and stored; retrying it is this system's job, not Stripe's.

Proving it

A property-based test takes one scenario's events, delivers them in random orders with every event sent twice, and checks that balances and payout statuses always end the same.

What it costs

An event that arrives early is applied late, by up to one sweep interval. In exchange the ledger never shows a state that did not happen: balances are non-negative at every commit.

And an event that can never succeed, such as a payout that truly exceeds the balance, surfaces as a stuck inbox entry with its error. That is exactly the signal a payout monitor exists to raise.

In the repository

NextTwenty-five payouts, a balance that covers ten