Load balancing and traffic

Hashing a key modulo the number of servers works perfectly until that number changes. With four servers and a fifth added, the modulus changes for every key, so roughly four keys in five map somewhere new. Every cache empties simultaneously, every request misses, and the database receives the entire working set as cold reads at the exact moment you were adding capacity because you were already under load.

Consistent hashing places both servers and keys on a ring of hash values, and a key belongs to the first server clockwise from it. Add a node and only the keys between it and its predecessor move, which is roughly one over n of the total rather than nearly all of it. Remove a node and its keys go to its successor, and nothing else is disturbed.

Plain rings are lumpy, because a handful of random positions rarely divide a circle evenly, and one server can end up owning a third of the space. Virtual nodes fix it: each physical server takes many positions on the ring, typically a hundred or more, so the law of large numbers does the balancing. It also gives you weighting for free, since a larger machine can simply be given more positions.

Two refinements are worth recognising. Bounded loads add a capacity limit per node, so a node that is already at its share passes the key onward rather than accepting it, which stops a hot region overwhelming one server. Rendezvous hashing reaches the same goal by a different route: hash the key with each server's name and pick the highest score, which needs no ring and handles weighting cleanly.

The property to hold on to is that this is a technique for making membership changes cheap, not for making distribution perfect. A single very hot key still lands on a single server whatever the scheme, and the answer to that is replication of that key or a small cache in front, which is a different problem with a different fix.

A health check answers a question the load balancer is about to act on, so what it asks decides what happens during a failure. A shallow check confirms the process is alive and listening, which is cheap and catches a crashed or hung process. A deep check confirms the instance can reach the dependencies it needs to serve a request, which catches far more and is where the danger is.

The danger is correlation, and it is a property of the design rather than bad luck. Every instance checks the same database, so when that database wobbles every instance fails its check in the same second, the balancer removes all of them, and a dependency that was merely slow becomes a total outage with no healthy targets left to receive traffic. The failure path amplified the failure, which is worse than having no failure path at all, because at least some requests would have succeeded.

The compromise that works is a deep check that degrades rather than fails. Require several consecutive failures before reporting unhealthy, so a single slow response does not remove an instance. Give the check its own short timeout, well inside the interval, so a hanging dependency does not make the check itself hang. Never let a dependency the request path does not need mark you down, which sounds obvious and is routinely violated by a check that pings six services because six were available to ping. And where the balancer supports it, use the setting that stops removing instances once too few remain healthy, which converts the correlated failure into degraded service rather than none.

Keep readiness and liveness separate, because they answer different questions and conflating them is how a slow dependency becomes a restart loop. Liveness asks whether to restart this process, so it should be shallow and almost never fail: a process that is running should say yes even while a dependency is down, because restarting it will not fix the database. Readiness asks whether to send it traffic, so it may legitimately say no while a cache warms or a migration finishes. Point liveness at a deep check and a database blip restarts the fleet, which loses the warm caches and the in-flight requests as well.

Connection draining is the other half and it is what makes deploys invisible. When an instance is removed, in-flight requests need time to finish before the process exits, so the balancer stops sending new work, waits for a grace period longer than a normal request, and only then lets it go. Without it a rolling deploy drops live traffic on every step, which shows up as a small error spike at each deploy that everybody eventually stops noticing. The sequence that works is to fail readiness first, keep serving for a few seconds while the balancer notices, then stop accepting and finish what is open.

Load balancing inside a region is a solved problem with well-understood algorithms. Getting a request to the right region in the first place is a different question with three common answers, and they differ in how quickly they can change their mind.

DNS-based routing hands out different addresses by geography or by health, which is simple and works everywhere. Its weakness is the one every DNS topic returns to: the answer is cached by resolvers and clients for a duration you request and do not control, so failing over means waiting for other people's caches to expire.

Anycast advertises the same address from many locations and lets the network choose, so a client reaches whichever site is closest in routing terms. Failover happens in the routing layer within seconds, no cache is involved, and the address never changes. This is how large CDNs and public DNS resolvers work, and it is why they can lose a location without anybody noticing.

The third option is a global load balancer at the provider: one address, health-checked backends in several regions, traffic steered by latency or by policy. It is the least work and it puts the provider's control plane on the critical path, which is a dependency worth naming rather than assuming.

Whichever routes the traffic, the harder problem sits underneath: data. Serving reads from a nearby region is straightforward; accepting writes in several is a consistency design, not a routing one. Most systems that describe themselves as multi-region are read-local and write-to-one, and being explicit about that is more honest than the label.