Real-time and push

Persistent connections are stateful, which breaks the assumption that makes web services easy: that any server can handle any request. A message for a user must reach the exact process holding that user's socket, and nothing in a normal load balancer helps with that, because the message did not arrive as a request from that user.

The standard answer is a pub/sub layer. Each gateway subscribes to the channels for the connections it holds, and a publish from anywhere fans out to whichever gateway is holding the socket. The sender no longer needs to know where anyone is connected, which is the property that makes the design scale. Slack and Discord both work this way.

The alternative, a shared map of user to server, looks simpler and ages badly: it needs updating on every connect and disconnect, it is wrong during the seconds after a crash, and every sender must read it before sending. It is a directory that must be perfectly accurate to be useful, which is a hard property to buy.

Capacity is a memory question rather than a CPU one. Each idle connection costs kernel buffers plus whatever your runtime allocates per connection, so the ceiling per machine is usually measured in tens or hundreds of thousands and is decided by that per-connection overhead. The classic C10K work, and the C10M writing that followed, is entirely about driving that number down: event loops instead of threads, careful buffer sizing, and avoiding per-connection allocations.

Deploys are the part that surprises people. Restarting a gateway drops every connection it holds, so a rolling deploy across ten nodes is ten thundering herds of reconnections, each one a handshake and an authentication. Reconnect with exponential backoff and jitter on the client, drain connections gradually rather than all at once, and expect the reconnect storm to be the largest load your authentication path ever sees.

Finally, resumption is what makes reconnection invisible. The client remembers the last message id it processed and asks for what followed; the server keeps a short buffer per channel to answer that. Without it, every reconnect is either a gap in the conversation or a full resynchronisation, and at scale the second one is its own outage.

Presence looks trivial in a specification and is among the most expensive features in a chat product. Every state change potentially notifies everyone who can see that user, so the cost tracks the social graph rather than the number of events. In a workspace of 10,000 people where everyone can see everyone, one person arriving in the morning is 10,000 notifications, and a morning is 10,000 of those.

The data is worthless within seconds, which is the property that makes it tractable. It lives in memory behind short expiries rather than in a database: a heartbeat every 30 seconds refreshes a key with a 45-second lifetime, and absence of a heartbeat is absence of a user. Nothing needs to be deleted, nothing needs to be reconciled after a crash, and a lost update corrects itself within one interval.

Fan-out is reduced by scoping rather than by cleverness. Only notify people who could actually observe the change: those with the conversation open, in the same channel, on the same screen. That turns a graph-sized problem into a viewport-sized one, and it is why presence is usually accurate for the handful of people you are looking at and stale for everyone else, which is exactly the right trade.

Batching does the rest. Collect changes over a second or two and send one update rather than twenty, coalesce repeated transitions so a flapping connection produces one event, and let the client interpolate. Typing indicators are throttled hardest of all, often to one event every few seconds per conversation, because they are the first thing worth sacrificing under load and nobody notices their absence for a moment.

The last piece is degradation. Under pressure, presence should stop first, then typing indicators, then read receipts, before anything touches message delivery. Deciding that order in advance is what makes the feature safe to ship, because presence is exactly the kind of feature that will otherwise consume the capacity that messages needed.