Quorums, PACELC and failure domains

CAP describes behaviour during a partition, which is a real but rare event. PACELC adds the branch that dominates every other day: else, when the network is healthy, you still trade latency against consistency on every single request. Both halves are decisions; only one of them is a decision you make thousands of times a second.

The else branch is concrete. Reading from the nearest replica is fast and possibly stale. Reading through a quorum is consistent and pays the round trips to reach it. Acknowledging a write locally is fast and risks losing it; waiting for a remote replica is durable and slower by the distance between them. No partition is required for any of those choices to exist.

The notation is worth reading properly, because it says two things. A system described as PA/EL chooses availability when partitioned and latency the rest of the time, which is Dynamo and Cassandra with their default settings. PC/EC chooses consistency in both, which is a system like a single-primary relational database with synchronous replication. Some are PC/EL: strict during a partition, fast when healthy, which is a legitimate and common combination.

It also explains a class of decision that CAP alone makes look arbitrary. When a database offers strongly consistent reads at higher cost and eventually consistent ones by default, that is the else branch exposed as an API parameter. The same design answering both is not a contradiction; it is a system that declined to make the trade on your behalf.

The habit worth forming is to state both halves when describing a system. During a partition, this refuses writes to these entities and accepts them for those. In normal operation, reads default to the nearest replica and these three paths pay for quorum. That is a specification. The letter on its own is a slogan.

And the geography question follows immediately, because latency in the else branch is set by distance. A quorum spanning continents pays the worst inter-region round trip on every write, which is why systems that need both properties shard so that each shard's quorum stays inside one region.

A cell is a complete, independent copy of the stack serving a subset of users: its own compute, its own database, its own cache. Nothing is shared, which is the entire property. A failure inside one cell has no path to the others, not because it is unlikely to spread but because there is no mechanism by which it could. That converts availability from a probability into arithmetic. Instead of arguing about how likely a total outage is, you decide in advance that the worst single failure affects one over n of your users. A bad deploy, a poison request, a corrupted cache, a runaway migration: each is contained to the cell it happened in, and the incident is a fraction rather than an event.

Deploys follow the same structure and become the main day-to-day benefit. Ship to one cell, watch it, then continue, which is a canary with a hard boundary rather than a percentage of traffic that shares a database with the other 95 per cent. Most bad changes are caught with one cell's users affected, and rolling back is a decision about one cell.

The routing layer is the part that has to be right, because it is shared by definition. It maps a user or tenant to a cell and must be simple enough to be nearly incapable of failing: a lookup, a hash, a static assignment, ideally cached at the edge and statically stable so it keeps working when its own control plane is unavailable. A clever, dynamic router is a single point of failure in front of an architecture built to have none.

The costs are real and worth stating. More infrastructure, because each cell needs its own everything and cannot pool spare capacity with its neighbours. Anything genuinely global, a report across all users, a search over everything, a migration, becomes a fan-out across cells with its own coordination. Cells also have to be sized, and a tenant that outgrows one is an awkward conversation.

AWS builds this way and publishes the reasoning, which is that a fault should have a bounded and known set of affected customers. That is the sentence worth keeping: not that failures are prevented, but that their extent is decided in advance rather than discovered during the incident.

In a request that fans out to many servers, the slowest response decides the total, so the distribution you care about is not each server's but the maximum across all of them. That maximum behaves very badly as the fan-out grows, and the arithmetic is the whole topic. Fan out to a hundred servers, each independently exceeding a second on one per cent of requests, and the chance that at least one does is one minus 0.99 to the hundredth, about 63 per cent. A one-in-a-hundred event at the component has become a two-in-three event at the request, which means each component's 99th percentile is roughly the request's median.

That reframes what to work on. Tuning the median of a dependency changes almost nothing for a fan-out request, because the median was never what you were waiting for. Cutting a dependency's 99.9th percentile changes a great deal. It is also why large systems set latency targets on their internal services at percentiles that look absurd in isolation: the caller's fan-out is what makes them reasonable.

The causes of a component tail are mostly not the component's code, which is why they resist the usual profiling. Shared resources contended by a neighbour on the same machine. A background task, compaction, garbage collection, log rotation, taking the CPU for a moment. Queueing behind a request that arrived first. Power and thermal management changing clock speed. A retry underneath you. Each is brief, uncorrelated with your request, and unavoidable, which is why the strategy is to tolerate the tail rather than to eliminate it.

Hedging tolerates it directly and cheaply. Send the request to one replica, and if it has not answered within a threshold set near the 95th percentile, send a duplicate to a second replica and take whichever returns first. Because only the slowest few per cent are ever duplicated, the extra load is a few per cent while the tail improvement is large. Dean and Barroso reported a case where hedging after a 10ms delay cut the 99.9th percentile of a thousand-server read from 1,800ms to 74ms for around two per cent more requests, which is the clearest available demonstration that the tail is a distribution problem rather than a capacity problem.

Tied requests are the refinement worth knowing. Send to two replicas immediately, and tell each which other replica has the request, so whichever dequeues it first cancels the other. That removes the waiting threshold entirely and costs less duplicated work than hedging, at the price of needing the replicas to talk to each other. The other mitigations are structural: many small partitions rather than few large ones, so a slow server's share can be moved elsewhere; and returning early with whatever has arrived, because a slightly incomplete answer inside the budget beats a complete one outside it.

Little's Law is the other half, and it connects the numbers you actually control: concurrency equals arrival rate multiplied by latency. It is the reason a tail is not merely a latency problem but a capacity problem in disguise. A service taking 100 requests a second at 50ms holds 5 in flight; the same service at 5 seconds holds 500, with nothing about the traffic or the code having changed. So when a dependency's tail lengthens, the in-flight work grows in proportion, the pool fills with requests that are waiting rather than working, and the queue runs away. That is why failing fast frees capacity and raising a timeout during an incident makes it worse, and it is why a concurrency limit is often a better control than a rate limit: it bounds the product rather than one of its terms.