Skip to content

WhatsApp

Delivery, ordering and encryption at billions of messages, with a server that is deliberately not allowed to read them.

01

A process per connection, and why Erlang

advanced

Messaging is not a request-response workload. Every user holds a connection open for as long as the app is running, because a message can arrive at any moment and polling for it would be both slow and expensive. So the primary resource is not processor time, it is connections, and the engineering problem is how cheap an idle connection can be made.

WhatsApp is well known for handling extraordinary numbers of connections per server, with published accounts describing a million or more on a single machine, and for running an unusually small engineering team while doing it. The choice that made both possible was Erlang, which is not a fashionable answer but is very nearly the correct one for this specific shape of problem.

Erlang's processes are not operating system threads. They are extremely lightweight, scheduled by the runtime, with tiny initial stacks that grow only if needed. That makes one process per connection viable, where one thread per connection would exhaust memory at a fraction of the count. The model is direct: a connection arrives, a process owns it, the process holds the session state, and when the user disconnects the process exits and everything it held is gone. No pooling, no multiplexing, no shared mutable state to reason about.

The second property matters as much and is less often mentioned. Erlang processes are isolated and fail independently: a crash takes down one connection's process and a supervisor restarts it, without touching the other million. In a language where connections share a heap, a bug triggered by one user's malformed input can take down everything on that machine.

There is also a deployment property that mattered for a small team. Erlang supports upgrading code in a running system, which for a service holding millions of long-lived connections is the difference between a routine deploy and disconnecting everybody.

Check yourself

Why does process isolation matter as much as process weight for this workload?

spawnwatchesregistersrecipient offlineClientsmillions, mostly idleAcceptorspawns per connectionConnection processowns session stateSupervisorrestarts on crashRoutingwhich node holds whomOffline queueuntil delivered
ClientEdge / CDNServiceData storeDashed = asynchronous
One process per connection, isolated and independently restartable
Evidence · 1 source
  1. 01
    WhatsApp engineering, 1 million is so 2011

    The published account of holding well over a million concurrent connections on a single machine, which is the figure the Erlang argument rests on.

Ask about this

Answers are generated and can be wrong. The topic above is the reviewed version.

02

Delivery: the ticks, and what each one actually promises

intermediate

The tick marks are a user interface for a distributed systems problem, and they are unusually honest about it. One tick means the server has the message. Two ticks mean the recipient's device has it. Two blue ticks mean the recipient's app has displayed it. Three separate facts, because in a distributed system they are genuinely three separate facts and collapsing them would be a lie.

The mechanics behind the first tick are what make the rest possible. The sender transmits, the server persists the message, and only then acknowledges. That ordering matters: acknowledging before persisting would mean a server crash silently loses a message the sender believes was sent. Persist, then acknowledge, and a crash before the acknowledgement causes a retry rather than a loss.

If the recipient is connected, the server pushes immediately. If not, the message waits in a per-recipient queue until they reconnect, which is the ordinary case rather than an exception: phones lose signal, go into tunnels, and run out of battery constantly. Once delivered and acknowledged, the server has no further need for it, and WhatsApp's design deliberately discards it at that point rather than retaining a copy.

Ordering is subtler than it appears. Messages from one sender to one recipient must arrive in the order they were sent, which a sequence number per conversation gives you. Ordering across different senders in a group is not achievable in any meaningful sense, because there is no global clock and two people genuinely can send at the same instant. What actually gets shipped is a consistent order rather than a true one, which is enough because nobody can tell the difference.

Every step needs idempotency. Retries are certain on mobile networks, so each message carries an identifier and a redelivery of one already seen is dropped rather than shown twice.

Check yourself

Why must the server persist a message before acknowledging it to the sender?

sendstoreone tick, after storingon reconnectdelivered, two ticksread, blue ticksSenderMessage servicepersist firstOffline queueper recipientDeliveryif connectedRecipient device
ClientServiceData storeDashed = asynchronous
Persist, then acknowledge. Each tick is a different fact.
Ask about this

Answers are generated and can be wrong. The topic above is the reviewed version.

03

End-to-end encryption, and designing so the server cannot help

advanced

WhatsApp uses the Signal Protocol, which combines an initial key agreement with a ratcheting scheme that derives a fresh key for every message. The consequence that matters architecturally is that the server routes ciphertext it cannot read, and that constraint shapes everything else in the system.

The two properties worth naming are forward secrecy and post-compromise security. Forward secrecy means compromising a key today does not decrypt yesterday's messages, because those keys were derived and discarded. Post-compromise security means the ratchet heals: after an attacker obtains a key, continued normal messaging derives new keys they cannot follow. Together they bound the damage of a compromise in both directions in time.

Sending to a device you have never messaged requires a key exchange, and both parties being online simultaneously is not a realistic assumption. So each client uploads a batch of pre-keys to the server in advance. A sender takes one, performs the agreement against it, and sends. The server distributes public key material without ever holding the private keys, which is the property that keeps it honest.

Groups do not encrypt once per group, because there is no shared secret the server could hold without being able to read everything. Each sender establishes a sender key distributed to each member over their existing pairwise encrypted sessions, so the message body is encrypted once and the key is delivered individually. Cost grows with membership, which is a real constraint on group size.

Multi-device is where this gets genuinely hard, and WhatsApp's answer since 2021 is that each device has its own identity and its own session with every contact. A message to a contact with four devices is four encryptions. The alternative, sharing one key between a person's devices, would be simpler and would mean compromising any one device compromises all of them.

The honest limitation is that encryption protects content, not metadata. The server necessarily knows who talked to whom and when, because it has to route the message, and that metadata is often as revealing as the text would be.

Check yourself

What does end-to-end encryption specifically not protect?

fetch a pre-keyciphertext per devicenecessarily knownSender deviceown identity keyPre-key storepublic material onlyMessage serviceroutes ciphertextRecipient phoneown sessionRecipient laptopown sessionMetadatawho and when
ClientData storeServiceDashed = asynchronous
Keys at the edges, ciphertext in the middle, metadata unavoidably visible
Evidence · 1 source
  1. 01
    WhatsApp, encryption overview technical white paper

    The Signal Protocol construction described here: pre-keys uploaded in advance for asynchronous session setup, sender keys for groups, and a separate identity and session per device.

Ask about this

Answers are generated and can be wrong. The topic above is the reviewed version.

04

Media: the message is not the file

intermediate

Sending a photo through the message path would be a mistake. Messages are small, ordered and latency-sensitive; media is large, unordered and throughput-sensitive. Putting a fifty megabyte video through the same channel as a text message means one upload blocks a conversation.

So media goes out of band. The sender encrypts the file locally, uploads the ciphertext to a blob store, and sends a message containing a reference and the key needed to decrypt it. The message stays small and the transfers are independent, which also means an upload can resume after a network drop without affecting anything else.

The encryption ordering is the important detail. The file is encrypted on the device before upload, so the blob store holds bytes it cannot interpret, exactly like the message service. If it were uploaded in the clear and encrypted server-side, the end-to-end property would be broken by the largest and most sensitive payloads in the system.

Forwarding is where the design pays off. The same encrypted blob can be referenced by many messages without re-uploading, which is why forwarding something to twenty people is instant while the original send was not. The key travels with each message; the bytes are stored once.

Thumbnails are the small detail that makes it feel fast. A low-resolution preview is small enough to embed directly in the message, so a recipient sees something immediately while the full file downloads. It is the same instinct as a progressive image: show something true as early as possible rather than nothing until everything is ready.

Check yourself

Why is the file encrypted on the device before upload rather than by the server?

upload encryptedsmall message withthumbnailreference and keydownloadnew reference, no re-uploadSenderencrypts locallyBlob storeciphertext onlyMessage servicereference plus keyRecipientfetches and decryptsForwarded copiessame blob reused
ClientData storeServiceDashed = asynchronous
Encrypt locally, upload once, reference many times
Ask about this

Answers are generated and can be wrong. The topic above is the reviewed version.

05

Groups: where the costs stop being linear

advanced

A one-to-one message is one encryption and one delivery. A group message to two hundred people, each with several devices, is hundreds of deliveries and a key distributed to every one of them. The cost per message grows with membership, and because the server cannot read the content it cannot do anything clever to reduce it.

The sender key scheme keeps the encryption cost sane. Rather than encrypting the message body separately for every member, the sender generates a sender key, distributes it once to each member over the existing pairwise sessions, and then encrypts each message once with it. Distribution is linear in membership; ongoing messaging is not.

Membership changes force key rotation, and the reason is a genuine security requirement rather than tidiness. When someone leaves, the remaining members must rotate, otherwise the departed member can still decrypt everything sent afterwards, having kept the old sender key. In an active group this means rotations are frequent, and each one is another round of distribution.

Delivery fan-out is the other half. Every message multiplies by members and again by devices per member, and the offline case makes it worse: a member offline for a week has a queue that must be retained and then delivered in a burst when they return. This is why group size limits exist. They are not arbitrary product decisions, they are where the fan-out arithmetic stops being affordable.

It is worth contrasting this with a broadcast channel, which looks similar and is not. A channel has one sender and many passive readers, so it can be built as fan-out on read with no per-recipient encryption at all. The moment every participant can also send, the costs change entirely.

Check yourself

Why must the sender key rotate when a member leaves a group?

once per memberencrypted onceper deviceif offlineSenderSender keygenerated onceKey distributionover pairwisesessionsMessage servicefans out ciphertextMember deviceseach decryptsOffline queuesper absent member
ClientServiceData storeDashed = asynchronous
Distribute the key once, then encrypt once per message
Ask about this

Answers are generated and can be wrong. The topic above is the reviewed version.