Rate limiting and resilience

A timeout is the number that decides how long you are willing to hold a resource for an answer that may never come, and most systems get it by accident. The default in many HTTP clients is no timeout at all, or a connect timeout with no read timeout, which is the same thing where it matters: a socket that opens and then goes quiet holds a worker forever.

Little's Law gives you the arithmetic and it is worth doing on paper before an incident does it for you. Concurrency equals arrival rate times latency. A service taking 100 requests a second at 50ms needs 5 concurrent workers. The same service at 30 seconds needs 3,000. Nothing about the traffic changed and nothing about the code changed; the only thing that moved was latency, and the pool that comfortably held 5 is now short by a factor of six hundred. This is why a slow dependency is more dangerous than a dead one: the dead one returns instantly and you handle it.

Timeouts have to decrease as you go down the call tree. If a browser gives up after 10 seconds, the API waiting 30 for a service that waits 30 for the database means every layer is working on a result nobody is waiting for, holding connections that live requests need. The rule is that each hop's timeout is shorter than its caller's, with room for a retry if the layer retries.

The better version of the rule is a deadline rather than a timeout: a wall-clock instant that travels with the request, so each hop passes on the time remaining rather than starting a fresh clock. gRPC does this natively, and it is why a deep call tree there degrades sensibly. Over HTTP you carry it yourself in a header and enforce it in a middleware. The difference shows up in the tail: with independent timeouts, four hops of 2 seconds each can spend 8 seconds serving a client that left after 3.

Choose the number from the latency distribution, not from a round figure. A timeout at the 99th percentile plus a margin cuts off the pathological tail without failing the merely slow. Setting it at the mean fails a fifth of good requests; setting it at ten times the p99 means it never fires and you have written a comment rather than a control. The instinct during an incident is to raise the timeout, and it reliably makes things worse: the calls that were failing now wait longer before failing, holding more resources for longer, and the failure spreads from one dependency to everything sharing that pool. Failing fast is what frees capacity, and it is what turns a broken dependency into a degraded page instead of a total outage.

A retry is the cheapest available fix for a transient failure and the fastest available way to turn a small problem into an outage. Both are true, and which one you get depends on three decisions: whether the call is safe to repeat, how long you wait between attempts, and whether the system as a whole has a budget for retrying at all.

Safety comes first. Retrying a non-idempotent call after a timeout is how a customer gets charged twice, because a timeout tells you nothing about whether the work happened. The fix is an idempotency key generated by the client, stored server side with the result, so a repeat returns the original outcome rather than performing the work again. Without that, the only correct retry policy for a payment is no retries.

Immediate retries are the wrong shape because the thing you are retrying is usually overload, and hitting it again straight away adds to exactly the load that caused the failure. Exponential backoff, doubling the wait each attempt with a ceiling, gives the dependency room to recover. But backoff alone leaves every client synchronised: they all failed at the same instant, so they all retry at the same instant, and the recovering service is hit by a wall.

Jitter is what breaks the synchronisation, and AWS published the arithmetic in 2015. Full jitter, sleeping a random duration between zero and the capped exponential interval, produced substantially less contention and less total work than plain exponential backoff in their simulations. It feels wrong to make the delay random and it is the single highest-value line in most retry implementations.

The failure that surprises people is amplification. Three attempts at each of four layers is not three times the load, it is up to 3^4, or 81 times, because every layer multiplies the layer below it. A dependency that is slightly unhealthy receives a load spike caused entirely by the reaction to its own slowness, which guarantees it stays unhealthy. Retry at one layer, ideally the one closest to the user, and let the others fail through.

The systemic answer is a retry budget rather than a per-call limit. Google's SRE practice caps retries at around 10% of requests over a window: below the cap, retries proceed; above it, they are refused, so a broadly failing dependency cannot be flooded by a fleet doing what each node individually considers reasonable. Pair it with servers that say which failures are worth retrying, since a 429 or a 503 with Retry-After is an invitation and a 400 is not, and never retry a call whose deadline has already passed.

A circuit breaker counts failures over a window and trips once they cross a threshold, then fails fast for a cooling period, then lets a single trial request through to test recovery. Closed, open, half-open. The half-open state is the part that matters: without it, the full load arrives the instant the timer expires, knocks the recovering dependency over again, and the breaker flaps between states while everyone watches the dashboard oscillate. What a breaker buys you is not error handling, it is capacity. When it is open, calls fail in microseconds instead of holding a worker for the timeout, so the pool stays available for the requests that can still be served. It converts a dependency's failure from something that consumes your service into something that returns an error from one part of your service.

The threshold should be a rate over a window with a minimum request count, not a raw count. Five failures means nothing at 10,000 requests a second and everything at ten a minute, and a breaker that trips on the first five errors after a quiet night will open on ordinary noise. The usual shape is at least twenty requests in the window and a failure rate above half.

A bulkhead solves the neighbouring problem: it gives a dependency its own bounded pool of connections or threads, so calls to it cannot consume the resources that every other request needs. The name is from ship compartments, and the point is the same, one flooded compartment does not sink the vessel. A breaker stops you waiting on a broken thing; a bulkhead stops one thing taking everything with it while it breaks. They pair, and neither replaces the other.

The modern criticism of hand-tuned breakers is that the numbers are guesses that age badly. Netflix built Hystrix, ran it at scale, and then retired it in favour of adaptive concurrency limits, which infer the safe level of in-flight work from observed latency rather than from constants a human chose in 2014. The pattern survived; the fixed thresholds did not, and the lesson worth taking is that a threshold set once will be wrong after the next capacity change.

Whatever trips has to have somewhere to go. A breaker that opens and returns a 500 has only made the failure faster, which is genuinely valuable, but the better outcome is a fallback: last known good data, a default, or the feature quietly absent from the page. That decision belongs at design time, per dependency, and it is the difference between an incident and a slightly worse page.

Not every dependency is essential, but a system only knows that if someone wrote it down. If recommendations are down, the product page should still render, with a cached set, a default set, or without that section at all. Absent an explicit decision, the default behaviour is that any dependency failure becomes a total failure, and that is a design choice made by omission rather than by anyone in particular.

The exercise is to classify each dependency as critical or optional, and to give every optional one a named fallback: last known good value, a static default, or hide it. This costs almost nothing at design time and is expensive to retrofit, because by then the call is buried three layers down inside a function whose contract says it returns a list of recommendations, not a list or nothing.

Load shedding is the same idea applied to your own capacity. Past a certain arrival rate, some requests will not be served, and the only question is whether you choose which ones or let a queue choose for you. Choosing means rejecting cheaply at the edge with a 503 and a Retry-After, ideally by priority, so a checkout still works while a report export waits. Not choosing means every request sits in a queue getting slower until they all time out together, which is the strictly worse outcome where nobody is served and you paid for the work anyway.

Queues need bounds and an eviction policy for the same reason. An unbounded queue during overload fills with requests that will have timed out by the time they are picked up, so the server spends its capacity producing answers nobody is waiting for. Facebook's answer was to switch the queue to LIFO under load with a CoDel-style controlled delay: serve the newest requests, which still have a chance of being useful, and discard the oldest, which almost certainly do not. It feels unfair and it is the correct behaviour.

Static stability is the other half. A system is statically stable when it keeps working with the data it already has while its control plane is unavailable: the caches keep serving, the routing keeps routing, and only changes stop. AWS designs this way deliberately, which is why running instances survive control plane incidents. It is a useful test to apply to your own design: if the configuration service is down for an hour, does traffic keep flowing, or does everything stop the moment a cache entry expires?

Finally, degradation has to be visible or it becomes a lie. If the page renders without personalisation, something needs to say so, in a log at minimum and often in the interface, or the next person to look will conclude the personalisation service is fine because the page looks normal. The failure is confined, not fixed, and the difference matters to whoever is on call.