Twitter and the feed

Fan-out on write has one failure mode and it is severe. An account with fifty million followers posts once, and that single action becomes fifty million writes. Post ten times in a day and it is five hundred million. The work is unbounded in the follower count, and a handful of accounts can saturate the infrastructure that everyone else depends on.

Worse, it is bursty in exactly the wrong way. Popular accounts post at moments of high general activity, so the fan-out spike lands while read traffic is also peaking. A queue absorbs some of it, but a large enough fan-out delays every other user's timeline behind it.

The answer is a hybrid, and it is what Twitter, Instagram and Facebook all independently arrived at. Ordinary accounts fan out on write, because their follower counts are small and the cost is trivial. Accounts above a threshold, commonly cited around ten thousand followers, do not fan out at all. Their posts are fetched at read time instead.

So a timeline read does two things and merges them: fetch the precomputed list, and separately fetch recent posts from the small number of large accounts this person follows. Merge by time, return the top slice. The pull side is bounded because nobody follows very many celebrities, and the push side is bounded because ordinary accounts have few followers. Each strategy is applied exactly where its cost is small.

The threshold itself is an operational dial rather than a principle. Too low and too many accounts are pulled, making every read do more work. Too high and the write spikes return. It also creates a boundary that has to be handled: an account crossing the threshold needs its existing followers' timelines to stay coherent while the strategy changes underneath them.

A reverse-chronological timeline is simple, predictable, and increasingly useless as the number of accounts someone follows grows. Follow two thousand accounts and the last hour contains more posts than anyone will read, so the question shifts from what is newest to what is worth showing.

Ranking changes the shape of the system in a way that is easy to underestimate. Chronological order can be maintained incrementally: a new post goes at the head, and nothing else moves. A ranked order can change for a post that already exists, because it gained engagement, or because the reader's interests shifted, or because the model was retrained. The timeline is no longer an append-only list.

The usual resolution is to keep fan-out producing a candidate set in time order, then rank that candidate set at read time. Retrieval stays cheap and incremental; scoring happens on a few hundred candidates rather than everything ever posted. This is the same retrieve-then-rank shape that search engines use, and for the same reason: scoring is too expensive to apply to the full corpus.

The costs are worth naming honestly. A ranked timeline is not reproducible, so two people cannot compare what they saw. Debugging becomes statistical rather than deterministic. And engagement-based ranking has a well-documented pull towards content that provokes, because provocation generates engagement, which means the objective function is a product decision with consequences rather than a technical parameter.

Underneath everything is a graph: who follows whom. It looks like a two-column table and behaves like nothing of the sort, because of how it is queried and how skewed it is.

Both directions are needed and they are wildly asymmetric. Fan-out asks for an account's followers, which for a large account is tens of millions of rows and is read on every post. A timeline asks who someone follows, which is usually hundreds. Storing one relationship and deriving the other means one of these queries is always expensive, so both directions are typically materialised and kept consistent.

The distribution is the real problem. Most accounts have a handful of followers, and a few have tens of millions. A single row's worth of relationship for one account and a hundred million for another cannot sit comfortably in the same partitioning scheme: shard by account and the shard holding a very large account is enormously hotter than its neighbours.

The graph also mutates constantly, and each mutation has consequences. A follow means the new follower's timeline is missing that account's history, so either it is backfilled or their timeline is simply thin until new posts arrive. An unfollow means existing entries are now wrong. Blocks and mutes add filtering that must be applied at read time, because applying them at write time would mean rewriting timelines whenever a block changes.

That last point is a useful general lesson. Some things must be evaluated at read time, not because it is cheaper but because the alternative is a rewrite of everything already written.

Timelines answer what the people I follow said. Search answers who said anything about this, which is a completely different access pattern over the same data, and it needs its own index rather than a query against the post store.

Real-time search has an awkward requirement: something posted seconds ago should be findable. Traditional inverted indexes are built in batches, because merging into a large index is expensive. The usual resolution is tiered: a small in-memory index absorbs the newest posts and is searched alongside larger, older segments that are rebuilt less often. Queries hit both and merge.

Trends are a different problem again, and the naive version is wrong in an obvious way once stated. The most frequent terms are not trending, they are common: the same ordinary words appear constantly. What is interesting is the deviation from the expected rate for that term, at that time, in that place. A term appearing a thousand times an hour is unremarkable if it usually appears nine hundred times, and extraordinary if it usually appears twice.

That framing makes trends a statistics problem rather than a counting one, and it brings the problems that come with statistics. Baselines have to be maintained per term and per region. Manipulation is a constant, since a group can coordinate to lift a term artificially. Time windows matter enormously: too short and noise dominates, too long and nothing ever looks sudden.