Observability

An average hides the tail, and the tail is what people complain about. A system averaging 100ms can be taking five seconds on one request in a hundred: the mean barely moves, and the hundredth user is the one who writes to support. p50 describes the typical experience, p95 and p99 describe the worst of it, and only the second pair predicts churn.

The tail is also structurally worse than it looks in a system made of many services, and the arithmetic is the most useful thing in this topic. If a request fans out to 100 servers and each has a 1% chance of taking over a second, the chance that at least one of them does is 1 minus 0.99 to the power of 100, which is about 63%. A one-in-a-hundred event at the component becomes a two-in-three event at the request. That is the result from Dean and Barroso's paper on tail latency, and it explains why large systems fight for the 99th percentile of their dependencies rather than the mean.

There is a trap on the way to percentiles that catches almost everyone: you cannot average them. The mean of each host's p99 is not the fleet's p99, and it is not any statistic at all. Percentiles have to be computed over the whole population, which is why metric systems store histograms with fixed buckets, or sketches such as t-digest, rather than the summary number each host computed for itself.

Choose the percentile from the number of chances a user gets to hit it. A page that makes 20 API calls gives the p95 twenty opportunities to appear, so roughly two thirds of page loads will contain at least one p95 request. For anything a user does repeatedly, p99 and p99.9 are the honest targets, and for a nightly batch job the mean is genuinely fine.

Measure at the edge as well as inside. Server-side latency excludes queueing before your process, DNS, connection setup, the mobile network and the time the browser spends rendering, and those can dominate. A service that is fast in its own dashboards and slow to its users usually has that gap, and finding it means measuring where the user is rather than where the code is.

Finally, watch the shape and not only the number. A bimodal distribution, fast on cache hits and slow on misses, has a p50 and a p99 that describe two different populations and a mean that describes neither. Histograms show that immediately; a single summary number never will.

An SLI is a measurement, such as the proportion of requests served successfully in under 300ms. An SLO is the target for that measurement over a window, say 99.9% over 30 days. The gap between the target and 100% is the error budget, and it is the most useful idea in the set because it turns reliability from a virtue into a quantity. The budget makes the trade explicit. Spend it shipping quickly. When it is exhausted, stop shipping and spend the time on stability instead. That replaces a recurring argument about whether features or reliability come first with a number both sides agreed to in advance, when nobody was under pressure and nobody was defending a decision they had already made.

The arithmetic is worth knowing because the nines are less intuitive than they look. 99.9% over 30 days is about 43 minutes of budget; 99.99% is about 4 minutes and 20 seconds, which is less time than most teams take to acknowledge a page. Each additional nine costs disproportionately more and buys less, and choosing one because it sounds serious is how a team ends up permanently over budget and ignoring the whole scheme.

The SLI has to be measured where the user is, and it has to describe something a user would recognise. The fraction of successful requests at the load balancer is a reasonable proxy; CPU utilisation is not an SLI at all, because no user has ever noticed it directly. The good ones are usually availability, latency, correctness and freshness, and the test for a candidate is whether a person could describe it in a sentence without using the word server.

Perfect reliability is the wrong target, and this is the part that sounds like heresy until you cost it. If the network between the user and you fails more often than your service does, the last nine you bought is invisible to everyone, and it was paid for with the features that were not shipped. An error budget that is never spent is a signal that you are shipping too slowly, and it should prompt exactly as much discussion as one that is exhausted.

None of this works without agreement about who owns the number. An SLO written by an infrastructure team and imposed on a product team becomes a stick; one agreed by both, with the budget policy written down before the first breach, becomes the thing that ends the argument. The mechanism is social as much as technical, which is why the SRE literature spends more pages on the policy than on the maths.

An alert is a claim that a human should stop what they are doing right now. Judged that way, most alerting rules do not qualify, and the cost of the ones that do not is not the interruption but the habit: a person who has ignored forty pages will ignore the forty-first, and that is the one that mattered.

The first rule is to alert on symptoms rather than causes. Users notice slow pages and failed requests; they do not notice a full disk, a restarted pod or a queue with 4,000 messages in it, unless one of those is making pages slow, in which case the symptom alert already fired. Cause-based rules multiply as fast as the system grows and go stale silently, because nothing tells you when a threshold stopped mattering.

What to measure for a symptom is well-trodden ground. Google's four golden signals are latency, traffic, errors and saturation. The RED method for request-driven services is rate, errors and duration; the USE method for resources is utilisation, saturation and errors. They overlap deliberately, and any of them is enough. Picking one and applying it consistently is worth more than picking the best one.

Alert on error budget burn rate rather than on a raw threshold, because a raw threshold either pages on a blip or misses a slow bleed. The SRE Workbook's approach is multi-window and multi-burn-rate: page when the recent burn is fast enough to exhaust a meaningful share of the budget quickly, for example 14.4 times the sustainable rate over an hour, which spends 2% of a 30-day budget in that hour, and confirm it with a longer window so a one-minute spike cannot page anyone. A slower burn, several times the sustainable rate over six hours, is a ticket rather than a page.

Every page needs a runbook link, an owner and an action. If the honest answer to what should I do is wait and see, it should not have woken anyone, and the fix is to make it a dashboard or a ticket. This is also the test that keeps an alerting system from growing without limit: nobody adds an alert they have to write a runbook for unless they mean it.

Finally, review pages the way you review code. Count them per week, name the ones that were not actionable, and delete or demote them in the same meeting. An alerting configuration is the only part of most systems that nobody ever deletes from, which is precisely why it ends up describing a system that stopped existing two architectures ago.

Observability costs scale with the questions you might ask, not with the questions you do ask, and the mechanism is cardinality. A metric is stored as one time series per distinct combination of its labels, so a request counter labelled by endpoint (50), status code (6) and region (3) is 900 series, which is nothing. Add customer id with 10,000 values and it is 9 million, and the monitoring system falls over before the service does.

The rule that follows is that labels are for values you would group by, and identifiers are not among them. User id, request id, session id, full URL paths with ids in them, and error messages containing a stack trace all belong in logs or traces, where the cost is per event rather than per distinct combination held in memory forever. The tell is a label whose set of values grows with your customer count.

High-cardinality questions are legitimate, which is why the answer is not simply do not do that. What is this specific customer seeing, why is this one request slow: those need per-event data. The design that works is metrics for the aggregate and cheap alerting, traces and structured logs for the detail, sampled hard, and an index that lets you find the events belonging to one id. Trying to make one store do both is how observability bills come to rival compute bills.

Sampling is the other lever, and it should be deliberate rather than emergent. Head sampling at a fixed rate is cheap and keeps a representative picture but loses the rare failure, which is the thing you wanted. Tail sampling keeps what was slow or wrong. A practical compromise is to keep a small uniform sample for baselines and everything anomalous on top, and to make the sampling rate visible so nobody computes a rate from sampled data and forgets to scale it.

Retention deserves the same scrutiny as volume, because the value of a log line falls off a cliff after about a week while its storage cost does not. Most of the questions asked of observability data are asked within hours. Keeping metrics for a year at low resolution and logs for two weeks at full detail costs a fraction of keeping everything for a year, and answers almost every real question. There is a failure mode worth naming: monitoring that takes the system down. An exporter that scrapes every series on every request, a logging call inside a tight loop, or a debug level left on in production can consume more resources than the work being observed. Instrumentation is code that runs on the hot path, and it deserves the same review as the rest of it.