Networking and the web

TCP guarantees delivery and order: lost packets are retransmitted, and the receiver holds everything after the gap until it is filled. UDP does neither. It sends a datagram and forgets it, and anything that arrives, arrives. That guarantee is not free, and the price is head-of-line blocking. One lost packet stalls everything queued behind it, which is exactly right for a file, where byte 400 is meaningless without byte 399, and useless for a live call, where a frame retransmitted 400ms later is worthless by the time it lands. Video calls, games and DNS run on UDP and handle loss themselves; nearly everything else takes the guarantee and is glad of it.

TCP also does congestion control, which is the part people forget when they reach for UDP. It probes for available capacity, backs off when it sees loss, and shares the path with other flows in a roughly fair way. Choosing UDP means either implementing that yourself or being the traffic that ruins the network for everyone else, and the internet works partly because most flows are well behaved.

There is a third cost worth knowing: connection setup. TCP's three-way handshake is a full round trip before any data, and TLS adds one or two more on top. UDP has no handshake at all, which is why DNS uses it for a query and a reply that would otherwise cost three exchanges to set up a connection carrying one small message.

QUIC is the interesting middle. It runs on UDP and rebuilds reliability, ordering and congestion control in user space, per stream rather than per connection, so a lost packet stalls only the stream it belonged to. It also folds the transport and TLS handshakes together, so a connection is established in one round trip and a resumed one in zero. That is TCP's guarantees without TCP's blocking, at the cost of being much newer and living in user space where it can be updated.

The rule of thumb is unchanged: take the guarantee unless stale data is worthless. If you find yourself adding sequence numbers, acknowledgements and retransmission on top of UDP, you are reimplementing TCP, and the version you write will be worse than the one in the kernel.

HTTP/1.1 carries one request at a time per connection. Pipelining is in the specification and is effectively unused, because a slow response blocks the ones behind it and proxies handled it badly, so browsers instead open around six connections per host and developers bundled files to fit more work through them.

That constraint produced an entire generation of workarounds: sprite sheets combining twenty icons into one image, inlined CSS, concatenated JavaScript, and domain sharding, which spread assets across asset1, asset2 and asset3 to get eighteen connections instead of six. Every one of them was a workaround for a protocol limit, and every one of them costs something: a sprite sheet invalidates entirely when one icon changes, and sharding multiplies DNS lookups and handshakes.

HTTP/2 multiplexes many streams over one connection, which removes the reason for all of it. It also compresses headers with HPACK, which matters more than it sounds when a request carries a kilobyte of cookies and the body is a hundred bytes, and it supports server push, which was so hard to use correctly that Chrome removed support for it.

What HTTP/2 does not fix is that it still runs on TCP. TCP delivers one ordered byte stream, so a single lost packet stalls every stream sharing that connection, which is worse than HTTP/1.1's six independent connections on a lossy network. On a good connection HTTP/2 wins comfortably; on a bad mobile link the comparison is closer than the version numbers suggest.

HTTP/3 moves to QUIC over UDP, where each stream is ordered independently, so loss affects one stream instead of all of them. It also merges the transport and TLS handshakes into one round trip, and connections are identified by a connection id rather than by the four-tuple of addresses and ports, so a phone moving from wifi to mobile data keeps its connection instead of starting again.

The practical advice is to stop carrying the workarounds forward. Bundling still helps a little, because compression works better across concatenated files and there is per-request overhead even when multiplexed, but the aggressive version hurts caching: one changed line invalidates a megabyte. Domain sharding is now actively harmful, since it defeats connection reuse and adds handshakes that HTTP/2 was designed to avoid.

DNS turns a name into an address, and the answer is cached at every layer between you and the client: the browser, the operating system, the corporate resolver, the ISP's resolver. Each holds it for as long as the TTL you published, which makes TTL the single most consequential number in a migration plan. Low TTL means fast changes and more lookups; high TTL means the reverse.

The catch is that your TTL is a request, not an instruction. Some resolvers clamp it to a minimum of their own, some ignore it, and some clients cache for the life of the process regardless. A migration planned around 60 seconds can still be sending traffic to the old address hours later, from a client nobody can identify, which is why the old address has to keep working rather than being decommissioned on schedule.

The folk explanation, that changes take hours to propagate down from the root servers, is wrong and worth unlearning. Nothing propagates: authoritative servers are updated immediately, and the delay is entirely caches holding the previous answer until it expires. That distinction matters because it tells you what to do, which is to lower the TTL well in advance rather than to wait and hope.

The practical technique follows. Days before a cutover, drop the TTL to 60 seconds and let the old, long TTL expire everywhere. Make the change. Keep the old address serving, ideally proxying to the new one, until traffic to it has stopped for longer than any plausible cache. Then raise the TTL again, because a permanently low TTL costs a lookup on every cold client and makes your resolver a hard dependency of every page load.

Better still, do not fail over in DNS at all. Point the name at something stable, a load balancer or an anycast address, and move traffic behind it. Then a failover takes effect in seconds for everyone, rather than in an interval decided by other people's caches. This is why every large system's public name resolves to an address that has not changed in years. One more property worth knowing: negative answers are cached too. If a name does not exist when someone looks it up, that absence is cached according to the SOA record's minimum, so creating a record after a client has already asked for it can leave that client failing for the duration. Create the record before you tell anyone to try it.

A CDN puts copies of your content in data centres near your users, so the round trips that dominate page load take tens of milliseconds instead of hundreds. Nothing about your origin got faster. The distance got shorter, and distance was the bill.

It helps even when nothing is cached, which surprises people. The connection is established with a nearby edge node, so the expensive handshakes happen over a short path, and the edge holds a warm, tuned connection back to your origin. A cache miss served through an edge is often meaningfully faster than the same miss served directly, purely from terminating TLS close to the user.

Static assets are the easy part, and the reason they are easy is content addressing: a hashed filename can be cached forever because a new build is a new URL. The interesting work is caching HTML and API responses, where you have to be deliberate about the cache key and about invalidation, which are the two things that go wrong and the two things nobody owns.

Cache keys are where hit ratios are won and lost. The key is the URL plus whatever Vary names, so an analytics parameter appended to a shared link produces a distinct entry per recipient, and Vary on a header with high cardinality shatters one entry into thousands. Normalising the key, stripping parameters that do not change the response, is routinely the largest single improvement available and costs nothing to run.

Modern edge platforms also run code at those locations, which changes what can be moved. Authentication checks, personalisation, redirects, A/B assignment and geographic routing can happen near the user rather than at origin, so a personalised page can still be served from a cached shell with the personal part filled in at the edge. That is the difference between a CDN as a file server and a CDN as part of the application.

The failure mode to respect is that an edge cache is very good at holding whatever you told it to hold. A wrong header applied by path rather than by response can pin the wrong content under the right URL for as long as you specified, and no purge reaches copies already held in browsers. Immutable caching is a promise about the URL, and it should only be made where the URL identifies the bytes.