Payments: charging a card without charging it twice

Everything in the previous topic makes retries certain. The network times out, the client gives up, the user presses the button again, the queue redelivers, the deploy restarts a worker mid-flight. A payments API that is not idempotent will double-charge, and the only question is how often.

The mechanism is a key chosen by the client, unique per logical operation rather than per attempt, sent with the request and stored server-side alongside the outcome. A repeat of the same key returns the stored result rather than doing the work again. That distinction, per operation and not per attempt, is where implementations go wrong: a key regenerated on retry makes every attempt a new payment, which is precisely the situation it was meant to prevent.

The concurrency is the hard half, because two retries can arrive at the same instant. Recording the key after the work completes lets both pass the check and both charge. Recording it before, with a unique constraint, means the second insert collides and can be refused, which is why the standard shape is to insert the key immediately in a pending state, do the work, then update it with the result. A concurrent duplicate gets a conflict, and a caller receiving that knows to retry rather than to assume failure.

Then there is the crash between charging and recording, which no single database can prevent because the charge happened somewhere else. This is why the stored record must be written in the same transaction as the local state change, and why the result of the remote call is recovered by asking the provider about the merchant's identifier rather than by guessing. A pending key older than a few minutes is stale and must be resolved by looking it up, not by retrying blindly.

Stripe stores idempotency keys for 24 hours, which is a deliberate boundary rather than an implementation limit: retries beyond that horizon are no longer the same operation in any meaningful sense, and a key reused a week later is treated as a new request. Choosing that window is a real design decision, and choosing to keep keys forever is a slow leak with a large table at the end of it.

A balance stored as a number on a row is a design that cannot answer the questions a payments system is asked. It says what the balance is and not how it got there, so a discrepancy has no explanation, a correction destroys the evidence, and two processes updating it race. Every serious money system stores movements rather than balances, and derives the balance by summing them.

Double entry is the discipline that makes those movements checkable. Every transaction writes at least two entries that sum to zero: money leaving one account and arriving in another, with fees and taxes as their own accounts rather than as adjustments. If the sum of all entries is not zero, something is wrong, and that check can run continuously rather than being noticed by a customer.

Entries are append only. A mistake is corrected by writing a compensating entry, not by editing history, which means the record of what happened and the record of what should have happened both survive. That is a requirement in regulated environments and it is also simply how you keep the ability to explain a balance to somebody who disagrees with it.

The performance question that always follows is whether summing entries is too slow, and the answer is that you keep a materialised balance and treat it as a cache of the truth rather than as the truth. It is recomputed and compared, and a mismatch is an alert rather than a silent correction, which is exactly the property a stored-balance design cannot offer.

The last piece is that a ledger entry is not a payment. The payment is a state machine with an external dependency; the ledger is an internal record of money that has moved. Keeping them separate means an authorisation that never captures leaves no ledger entry, a refund is two entries rather than a subtraction, and a dispute writes entries of its own. Collapsing the two produces a ledger that has to model card network semantics, which is where the design stops being explainable.

Plenty of what happens to a payment happens after the response. A bank authorisation can be reversed. A dispute arrives weeks later. A bank transfer that looked successful can fail days afterwards. A subscription renews on a schedule nobody is watching. So the merchant needs a channel for outcomes that were not available at request time, which is what webhooks are for.

That makes webhook handling a core part of a payments integration rather than an optional convenience, and it inherits every property of delivery over an unreliable network. Events can arrive twice, out of order, or after a long delay while a receiver was down. Handlers must be idempotent by event id, must tolerate an older event arriving after a newer one, and must not assume that the absence of an event means the absence of the outcome.

The reconciliation of local state with the provider's is where the design earns its keep. A merchant's database and a payment provider's are two systems with no shared transaction, so they will disagree at some point. Treating the webhook as the only source of updates makes a missed delivery permanent, which is why a periodic sweep that asks the provider about anything in a non-final state is a required component and not a belt-and-braces addition.

Ordering deserves a specific mention because payments make it visible. A charge succeeded event and a charge refunded event delivered out of order will, in a naive handler, leave a refunded payment marked as succeeded. Handlers that check the state transition rather than applying the event blindly, or that fetch the current object rather than trusting the payload, avoid a whole family of these bugs.

The user-facing consequence is that the interface has to be honest about pending. Telling somebody their payment succeeded when it has only been accepted produces the worst possible support conversation later. Showing a state that can still change, and notifying when it does, is both more accurate and, in practice, less alarming than a success that quietly reverses.

Declines are not errors in the usual sense. A hard decline means the card is closed, stolen or invalid and will never work; retrying is pointless and, at volume, damages the merchant's standing with the networks. A soft decline means insufficient funds or a temporary block, and a retry in a few days has a genuine chance. Treating the two identically is the most common expensive mistake in subscription billing.

The reason to care is that the response codes are the only signal, and they are coarse. Issuers deliberately do not explain much, partly for fraud reasons, so a system has to map a small set of codes into a policy: retry, retry later with a different strategy, or stop and ask the customer. Getting that mapping wrong shows up as either lost revenue or an unhappy conversation with a payment provider about retry ratios.

For recurring payments, the practice built on this is dunning: a schedule of retries spread over days, combined with messages to the customer, because many failures are fixed by the cardholder rather than by the system. The retries are timed to when they are more likely to succeed, such as after a typical payday, and the whole sequence has an end, after which the subscription is cancelled rather than retried indefinitely.

Network-level failures need the opposite instinct. Where a decline is an answer, a timeout is the absence of one, and retrying a timeout without an idempotency key is how double charges happen. So the two paths must be distinguished in code: a decline is a business outcome to be recorded and acted on, a timeout is an unknown to be resolved by lookup before anything else is attempted.

Card details also expire and change, which is a background failure mode with a documented remedy: account updater services that keep stored credentials current, and network tokens that survive a card being reissued. For a business with stored cards, that machinery recovers a meaningful share of otherwise-lost revenue, and it is invisible to anyone who has not been told it exists.

At the end of every day, three records exist of the same money: what the merchant's system believes, what the payment provider reports, and what the bank actually moved. They will not agree exactly, and the job of reconciliation is to explain every difference rather than to hope there are none.

Most differences are timing rather than error. A capture on one side of midnight and a settlement on the other, a refund issued today and settled in three days, a payout batching Friday and Saturday together. A reconciliation process that cannot express in flight will produce alarming numbers every single day and will therefore be ignored, which is worse than not having one.

The differences that are not timing fall into a small set: a payment recorded locally that the provider has never heard of, usually a request that timed out and was never resolved; a provider payment with no local record, usually a webhook that was missed and a sweep that never ran; and amount mismatches, usually fees or currency conversion applied somewhere the local model does not represent.

Which is why the reconciliation is also the test of everything upstream. A system with correct idempotency, a working sweep and an honest ledger reconciles with a small, explainable set of in-flight items. A system without them reconciles with a list of mysteries, and the mysteries are the same bugs described in the earlier topics arriving in aggregate at the end of the month.

The operational shape that works is a daily automated match with a named owner for the exceptions, and a small tolerance that is a deliberate decision rather than an accident. Rounding in currency conversion is real, and pretending otherwise produces an exception queue nobody can clear. What matters is that the tolerance is written down, monitored for drift, and never used to absorb a difference that is growing.