Concurrency and operating systems

A race condition is a correctness bug that depends on timing, which is why it passes every test on a laptop and appears in production at a rate proportional to traffic. Read-modify-write is the classic shape: two requests read the same value, both compute from it, and one result is silently lost. Nothing errors, nothing logs, and the total is simply wrong.

Locks prevent it by serialising access, and they are the first instinct because they are the most obvious. The costs are contention, since everyone waits their turn on the hot row, and deadlock, when two paths take the same two locks in opposite orders and each waits for the other forever. Deadlock is preventable by always acquiring locks in a fixed global order, which is a rule that has to be written down somewhere because it cannot be enforced by the type system.

The better fix is usually to remove the gap rather than guard it. An atomic increment does the read and the write as one operation the database cannot interleave. A conditional update, set stock to 0 where stock is currently 1, fails cleanly when someone else got there first, and the caller retries or reports a conflict. Both turn a timing bug into a visible outcome.

That conditional update is optimistic concurrency control, and the general form is a version column: read the row and its version, write with a condition that the version is unchanged, and bump it. Zero locks are held between the read and the write, which is the point, because the user's thinking time is not something to hold a lock across. It is the right default when conflicts are rare, and the wrong one when they are common, because every conflict costs a wasted round trip.

Pessimistic locking is the other half of that pair: take the lock first, hold it for the duration, and make everyone else wait. SELECT FOR UPDATE is the usual form. It is right when conflicts are frequent enough that retrying is more expensive than queueing, and it needs a timeout, because a lock held by a process that died is a lock held forever unless something reclaims it.

Distributed locks deserve suspicion. Once the lock lives in another system, a process can be paused by garbage collection, lose its lease, and resume believing it still holds it, while another process holds it for real. Fencing tokens are the standard mitigation: the lock hands out an increasing number, the resource rejects any write carrying a number lower than the highest it has seen, and the zombie's late write is refused. If you can restructure the problem to avoid needing the lock, that is almost always cheaper than getting this right.

An idempotent operation can run repeatedly with the same result as running once. This matters because networks make retries unavoidable, and a timeout tells you nothing at all about whether the work happened: the request may never have arrived, or it may have completed and the response been lost on the way back.

The standard approach is an idempotency key supplied by the client, unique per logical operation rather than per attempt. The server records the key alongside its result, and returns the stored result if the key comes back. Without it, a retried payment is simply a second payment.

The subtlety is what you store and when. Recording the key before doing the work means a crash mid-operation leaves a key with no result and the retry is refused forever. Recording it after means two concurrent retries both pass the check. The usual answer is to insert the key immediately with a pending state, under a unique constraint so the second attempt collides, then update it with the result. A pending key older than a few minutes is stale and can be reclaimed.

Not everything needs a key. GET, PUT and DELETE are idempotent by their own semantics, and an update that sets an absolute value is naturally safe to repeat where one that increments is not. The key is for the operations that are neither, which in practice means anything that creates something or moves money.

When a system accepts work faster than it can finish it, the backlog grows, and there is no version of that story with a happy ending. Queues fill, memory climbs, latency climbs, and the failure arrives at the worst possible moment, which is later and larger than the failure you would have had if you had simply said no at the start. Backpressure is the mechanism for saying no: refusing or slowing intake when what is downstream cannot keep up. Bounded queues, load shedding, rejecting early with a 503 and a Retry-After. It feels like worse behaviour and it is better behaviour, because a rejection is a signal the caller can act on while a growing backlog is information nobody has.

TCP is the reference implementation and it is worth studying because everyone already depends on it. The receiver advertises a window, the amount it is prepared to accept; when the application stops reading, the window shrinks to zero and the sender stops sending. No messages are lost, no buffer grows without bound, and the pressure propagates back to the source through nothing more than an advertised number. Every good backpressure design is a version of that.

In an application it usually means bounding the things that are unbounded by default. A queue with a maximum depth that rejects rather than growing. A connection pool with a fixed size and a short wait, so a slow database produces immediate errors rather than a thousand threads waiting on a pool. A concurrency limit per dependency. Each of these converts an invisible accumulation into an error with a name.

The subtle failure is backpressure that stops at a boundary. A service that rejects work correctly, in front of a queue that accepts everything, has moved the unbounded buffer one step upstream and made it someone else's surprise. The property you want holds end to end: if the slowest component in a chain can only manage 100 a second, the intake at the front should be admitting roughly 100 a second, not 10,000 into a buffer.

Where it cannot propagate, drop deliberately and say so. For a live telemetry stream or a metrics pipeline, the right answer under overload is to shed the excess and record how much was shed, because stale telemetry has no value and a growing buffer of it has negative value. That decision belongs in the design with a number attached, rather than emerging at 3am as an out-of-memory kill.