Netflix

For years the industry used one fixed bitrate ladder for everything: 1080p at some bitrate, 720p at a lower one, and so on down, identical for every title. This is obviously wrong the moment you say it out loud. An animated film with flat colour and clean lines needs far fewer bits to look perfect than a handheld night scene full of grain and motion, yet both were given the same allocation. One was wasting bandwidth and the other was being starved of it.

Netflix replaced this with per-title encoding: analyse the content, then build a ladder specific to it. The complex title gets more bits where it needs them; the simple one gets a ladder that tops out lower because there is nothing more to represent. The measure used to decide is VMAF, a perceptual quality metric Netflix developed and open-sourced, which correlates with what people actually report seeing rather than with mathematical error like PSNR.

Then they went further, to per-shot. A film is not uniform: within one title there are dialogue scenes that are trivial to encode and action sequences that are not. Dynamic Optimizer detects shot boundaries, the hard cuts where one camera setup ends and another begins, and optimises each shot independently, choosing the resolution and quantiser that maximise VMAF for the bits available.

The reported savings are large. Netflix published reductions of roughly 28% for x264, 38% for libvpx VP9 and 34% for x265 at equal quality, with a further 10 to 15% on top of per-title encoding at the same VMAF. They enabled it for selected titles in 2018 and broadly for 4K in 2020.

It is worth being precise about what is being traded. This is enormous extra compute at encode time, spent once per title, to save bandwidth on every single playback of that title forever. For a catalogue watched billions of times, that trade is not close.

Once the ladder exists, something has to choose which rung to play, and that decision belongs on the client, not the server. The client is the only party that knows what is actually happening: how full its buffer is, how fast the last few segments arrived, whether the device is thermally throttled, whether the user just seeked.

The mechanism is simple. Video is cut into segments of a few seconds, each available at every rung of the ladder. The player keeps a buffer of upcoming segments and, for each new request, picks a rung. Ask for too high a rung and the segment arrives late, the buffer drains, and playback stalls, which is the single worst thing a video player can do. Ask for too low a rung and the picture is soft when it did not need to be.

Naive algorithms estimate throughput from recent downloads and pick the highest rung that fits. This behaves badly in exactly the situations that matter, because throughput estimates are noisy and a brief dip triggers a visible quality drop that was not necessary. Better approaches use buffer occupancy as the primary signal: a full buffer is direct evidence that you can afford to be ambitious, and it is a measurement rather than a prediction.

There is a startup problem too, and it is the one users judge you on. At the moment of pressing play there is no buffer and no throughput history, so the player must guess. Guess high and the first segment is slow, which shows as a spinner. Guess low and playback starts quickly but looks poor for the first few seconds. Most players start conservatively and climb, because time to first frame is what people notice and remember.

Content licensing obliges Netflix to protect the video, and protection means encryption plus a licence server that decides who gets a key. The complication is that there is no single DRM system: Widevine covers Android and Chrome, PlayReady covers Windows and Xbox, and FairPlay covers Apple devices. Three ecosystems, three key formats, three sets of rules.

Encrypting the same title three times would triple storage and triple the encoding cost, which is why Common Encryption exists. The video is encrypted once with one key, and each DRM system is given a way to deliver that same key to its own devices. One set of files, three licence paths, which turns a threefold storage problem into a licence-server problem.

The licence request is where the real decisions happen, and they are business decisions rather than cryptographic ones: whether this account is entitled to this title in this country, how many concurrent streams the plan allows, whether the device is permitted at this resolution. The last one is not arbitrary. Studio agreements commonly tie higher resolutions to stronger protection, which is why the same title plays at 4K on a certified television and at a lower resolution in a browser on the same account.

On the web this is mediated by Encrypted Media Extensions, a browser API that hands encrypted segments and licence responses to a Content Decryption Module the page cannot see inside. The browser deliberately gives the page no access to the decrypted frames, which is the entire point.

Netflix is a large share of internet traffic at peak. Buying that from a commercial CDN would be enormously expensive, and it would still put the bytes on the wrong side of the congested links between networks.

So Netflix builds its own appliances and gives them to ISPs at no cost, installed inside the ISP's own network. The ISP supplies power, space and connectivity. Netflix supplies the hardware and operates it. By the time you press play, the video is already past the peering links that would otherwise be the bottleneck, sitting on a machine inside the network you are a customer of.

The published hardware makes the trade concrete. Storage appliances are 2U servers with up to 120TB of raw storage and around 200Gbps of operational throughput, connected by 6x10Gbps or up to 2x100Gbps. There is also a smaller global appliance with up to 60TB and around 80Gbps, aimed at smaller partners and emerging markets. Netflix states that Open Connect serves 100% of its video traffic, and that close to 90% of that traffic globally is delivered over direct connections between Open Connect and the residential ISPs its members use.

Content arrives on those appliances during off-peak hours, chosen by predictive models of what each region will want. This is the part that makes the whole thing work: the catalogue is finite, demand is forecastable, and yesterday's data tells you a great deal about tomorrow's viewing. Popularity is not uniform, so an appliance does not need the whole catalogue to serve almost all requests locally.

The appliances do not decide who they serve. They are directed caches: an OCA only serves clients whose prefixes the ISP has advertised to it over BGP, so the ISP retains control of what traffic lands where, which is a precondition for any ISP agreeing to host equipment at all.

Having thousands of appliances is useless without a good answer to the question of which one a given client should use, and the answer changes constantly. An appliance may be near you and busy, or slightly further and idle. It may hold the title you asked for, or not. It may have been healthy a minute ago and not now.

So the choice is made per session at playback time, not by DNS and not by anycast. When the client asks to play something, the control plane returns a manifest that already contains the URLs it should use, ranked, with fallbacks. This is a deliberate departure from how most CDNs steer traffic, and the reason is control: DNS-based steering is coarse, cached at resolvers beyond your control, and cannot see whether the specific title is present on the specific box.

The ranking considers proximity in network terms rather than geographic ones, current load on each candidate, health, and whether the appliance actually holds the title. Because the manifest carries fallbacks, a client that starts failing can move without another round trip to the control plane, which matters because the control plane is the thing you least want in the path of a recovery.

Note the split this creates. The control plane runs in the cloud and handles authentication, entitlement, and steering, all small requests. The data plane is the appliances, and it handles essentially all of the bytes. The two scale independently and fail independently, which is the property that makes the design robust.

The backend is hundreds of services, and a single home page assembles rows from many of them. The arithmetic here is unforgiving and worth doing explicitly. If each of two hundred dependencies is independently available 99.9% of the time, the probability that all two hundred are healthy simultaneously is 0.999 raised to the power of 200, which is about 82%. Roughly one page load in five would fail if every dependency were treated as required.

That number is the whole argument. At this many dependencies, you cannot buy your way out with better uptime per service, because the compounding beats you. The only viable answer is to make most dependencies optional: render the page with whatever is available, and degrade the parts that are not. If personalisation is unavailable, show a sensible default row rather than an error page.

The tooling exists to make that the default rather than an aspiration. Zuul is the gateway at the edge, rewritten on Netty for asynchronous non-blocking I/O to hold very large numbers of connections. Eureka handles service discovery. Hystrix, now retired in favour of newer libraries, popularised the circuit breaker and, more importantly, the idea that every remote call should have a declared fallback.

Chaos engineering came from this problem. A fallback that has never been exercised is a hypothesis, not a safety net, and the only way to find out whether it works is to break the dependency and watch. Doing that deliberately, in production, during working hours, is far better than discovering the answer at three in the morning.

EVCache is the distributed caching layer, built on memcached, replicated within a region across availability zones and also across regions. Reported throughput is on the order of hundreds of millions of operations per second, holding things like viewing history, session metadata and precomputed personalisation.

The interesting decision is not the cache itself but how invalidation is handled, because invalidation across regions is where most designs quietly become distributed transactions. A delete that must succeed in every region before a write is acknowledged is a coordination protocol, and it will fail exactly when the network between regions is already degraded, which is to say at the worst possible moment.

Netflix leans instead on short time to live values and versioned keys. A write bumps a version, so subsequent reads look for a key that only the new value occupies. Old entries are not deleted; they become unreachable and then expire on their own. Nothing needs to be coordinated, and a region that is temporarily unreachable simply serves slightly stale data until its entries age out.

Cross-region replication also exists for a reason that is easy to miss: it is not only about read latency, it is about failover. If a region is evacuated and its traffic lands elsewhere, the receiving region needs warm caches. A cold cache under a doubled load is its own outage, because every miss becomes an origin request at exactly the moment origin is busiest.

Netflix runs multiple AWS regions active-active, serving members from more than one at the same time rather than keeping a standby that has never taken real traffic. The reason for active-active over active-passive is not capacity, it is confidence: a standby region is a claim about what would happen, and claims about untested code paths are usually wrong.

Failover is therefore a traffic shift rather than a promotion. If a region degrades, its traffic is moved to the others, and because those regions were already serving, there is no cold start of anything, no first-time-ever code path, no configuration that was correct a year ago.

They rehearse it. Chaos Kong is the exercise that evacuates an entire region, and it is run as a matter of routine rather than as a drill after an incident. Published accounts describe keeping traffic out of a region for over twenty-four hours before shifting it gradually back to an even split, and shifting more slowly than an emergency would require, specifically so services can scale up and caches can warm. That detail is the interesting one: they discovered that the speed of the shift is itself a failure mode.

The write path is the genuinely hard part and it is worth being honest about it. Reads are easy to serve from anywhere. Writes accepted in two regions at once need either a partitioning scheme that keeps a given member's writes in one place, or eventual convergence with a defined conflict rule. There is no version of this that is simultaneously fast, globally consistent and available during a partition, which is the practical face of the CAP theorem rather than the textbook one.

Almost every row on the home page is chosen, ordered and titled for the member looking at it, and even the artwork shown for a title can differ between members. That is a great deal of computation to attach to a page load, and the design answer is that most of it is not attached to the page load at all.

The heavy work happens offline. Models are trained on viewing behaviour in batch, and candidate rows are precomputed and written to a store the serving path can read quickly, which is where EVCache earns its place. At request time the service is mostly assembling and lightly reranking things that were computed earlier. This is what makes a personalised page affordable.

The event pipeline feeding this is substantial in its own right: play events, pause events, searches, impressions of rows that were never clicked, all streamed and processed continuously. The negative signals matter as much as the positive ones, because a row shown many times and never clicked is evidence.

There is a cold start problem that no amount of infrastructure solves. A new member has no history, so the first session is populated from what they told you during signup, from what is popular in their region, and from the device they are on. It gets better quickly, and the honest framing is that the first session is the worst one they will ever have.

It is worth naming the failure mode too. Feedback loops are real: recommending what was clicked yesterday makes it more clickable today, and a system optimised only for immediate engagement will narrow what people see. This is why the objective being optimised matters more than the model, and why measuring long-term retention rather than the next click is the harder and more honest choice.