Design a payment system (correctness & idempotency)
Every other case study could tolerate a little staleness or loss. Payments cannot: charging a customer twice or losing a successful charge is unacceptable. This is the design round where correctness beats scale, and it's built to test the concepts the others let you wave away — exactly-once semantics, idempotency, consistency, and reconciliation. It's also where the durable interview word idempotency finally gets its full treatment.
📖 Walk me through it — plain English
The whole problem comes from one ugly fact: networks lose messages. A user taps "Pay $50." The request reaches your server, the charge succeeds — and then the response back to the phone gets lost. The phone, seeing no answer, retries. Now the same payment arrives twice. If you naively process it, you charge $100. The internet guarantees this will happen at scale; your design has to make a duplicate request harmless.
The fix is idempotency: making an operation safe to repeat, so doing it twice has the same effect as doing it once. The mechanism is an idempotency key — the client generates a unique ID for the intent ("this specific pay-$50 attempt") and sends it with every retry. The server remembers keys it has already completed: if a request arrives with a key it has seen, it returns the stored result of the first attempt instead of charging again. Same intent, one charge, no matter how many times the message is delivered.
The other half is that a payment touches more than one place — your balance, the customer's, a ledger, an external bank — and those must move together or not at all. You can't use one database transaction across systems you don't control, so you use careful ordering, a durable log of every state change (a ledger), and a background reconciliation job that compares your records against the bank's and flags any mismatch. The honest senior framing: you make double-charges nearly impossible, and you build the safety net that catches the rare case anyway.
Step 1 · Idempotency, the load-bearing idea
An operation is idempotent if applying it multiple times has the same effect as applying it once. Reading a value is naturally idempotent; "add $50" is not (twice = $100). The job is to make "charge $50" behave idempotently using a key:
- Client generates key pay_abc123 and sends "charge $50, key=pay_abc123".
- Server checks an idempotency table for pay_abc123 → not found. It inserts the key as "in progress" (a unique constraint makes this atomic), performs the charge, stores the result against the key, returns success.
- The success response is lost; the client retries with the same key pay_abc123.
- Server checks the table → found, already completed. It returns the stored result. No second charge.
The unique constraint on the key is what makes step 2 safe even if two retries race in at the exact same moment — the database lets exactly one insert win, and the loser is treated as a duplicate. That's the senior detail: idempotency leans on a uniqueness guarantee you already trust.
Step 2 · The ledger: never mutate, only append
Don't store a balance as a single number you overwrite — if an update is lost or doubled you can never reconstruct the truth. Instead keep a ledger: an append-only log of immutable entries ("+$50 from card X to merchant Y, at time T, ref pay_abc123"). The balance is the sum of entries, not a field. This is double-entry bookkeeping, centuries old, and it gives you an audit trail, the ability to replay history, and a natural way to detect tampering. Interviewers love hearing "append-only ledger, balance is derived" because it shows you value auditability over convenience.
Step 3 · Coordinating across services you don't own
A charge spans your service, a payment processor (Stripe/bank), and your ledger. You can't wrap an external bank call in your database transaction, so you sequence it carefully and make each step recoverable:
- Record intent first. Write a "pending" ledger entry + idempotency key before calling the bank, so a crash mid-flight leaves a record you can resume from, not a silent gap.
- Call the external processor with the same idempotency key. Good processors are themselves idempotent on that key, so your retry doesn't double-charge at their end either.
- On confirmation, append a "completed" entry; on failure, a "failed" one. The state machine (pending → completed/failed) is explicit and never skips.
- Use the saga pattern for multi-step flows: if a later step fails, run compensating actions (a refund entry) to undo earlier ones — because you can't roll back a real bank transfer, you offset it.
This is where you explicitly choose strong consistency over availability: if you can't be sure a charge succeeded, you fail closed (don't deliver the goods) rather than guess. That's the opposite of the social-feed choices earlier — and saying so out loud shows you pick consistency per the stakes.
Step 4 · Reconciliation: the safety net
Even with all the above, reality drifts — a webhook is missed, a timeout hides a success. So a periodic reconciliation job pulls the processor's record of truth and compares it line-by-line against your ledger, flagging any charge that exists on one side but not the other for human review or automated repair. Senior engineers assume their happy path is imperfect and build the reconciliation that catches what slips through. Naming this job is a strong closing signal.
Pitfalls
Relying on the network never retrying. It will. Without a key, a lost response becomes a double charge — the single most important thing to get right.
Overwriting a single number loses history and hides errors. Append-only ledger; derive the balance. Auditability is non-negotiable for money.
Guessing a charge succeeded to keep the flow moving. For money you fail closed — strong consistency wins over availability when the stakes are financial.
Trusting the happy path. Distributed money flows drift; without a job comparing your ledger to the processor's, mismatches go unnoticed until a customer complains.
Takeaway: payments is the round where correctness outranks scale. Make charges idempotent with a client-generated key plus a unique constraint, so retries can't double-charge. Keep an append-only ledger and derive balances for a full audit trail. Coordinate across external systems by recording intent first, passing the idempotency key through, and using sagas with compensating actions instead of impossible cross-system rollbacks. Choose strong consistency — fail closed — because the stakes are money, and run a reconciliation job as the safety net. This is the inverse of every staleness-tolerant design before it, and naming that contrast is the signal.
→ Going deeper: idempotency and retries trace back to the failure thinking in the operational layer; the consistency choice is the strong-vs-eventual axis from scaling primitives; and the "fail closed, defend the stakes" stance is a CBW decision under the highest stakes in the guide.