Queues and asynchronous work

At-most-once may drop messages. At-least-once may deliver twice. Exactly-once is what everyone wants, and end to end in a distributed system it is not achievable, because the acknowledgement itself can be lost and the sender cannot tell a lost ack from a lost message.

What gets sold as exactly-once is at-least-once delivery plus idempotent processing, so a duplicate has no additional effect. Kafka's exactly-once semantics are real but scoped: they cover reading from a topic, writing to a topic, and committing the offset, as one atomic unit inside Kafka. The moment your handler calls Stripe, that guarantee has left the building.

The rate is the part worth understanding, and it is not a single number anyone can quote across brokers. In steady state duplicates are rare enough that a system can run for weeks without one being noticed. They do not arrive uniformly: they cluster during a consumer group rebalance, a partition, a failover or a deploy that restarts a worker mid-batch, which is precisely when everything else is also going wrong and nobody has the attention to reason about them. So the number to design against is not the average rate but the burst, and the only design that survives a burst is a handler where a repeat costs nothing.

Idempotency in practice is a table of processed message ids with a unique constraint, checked inside the same transaction as the work. Insert the id, do the work, commit. A duplicate hits the constraint and rolls back having done nothing. Keep the ids for longer than your broker's maximum redelivery window, and remember that a natural key from the payload, an order id, is often better than the broker's message id, which changes on republish.

Ordering guarantees are usually per partition, not global. Kafka orders within a partition; across partitions there is no order at all, and a consumer reading two partitions sees them interleaved arbitrarily.

Keying by entity is what makes this workable. Hash the user id, or the order id, and every event for that entity lands in the same partition and arrives in the order it was produced. That is almost always the ordering you actually needed: nobody cares whether user A's update preceded user B's, but everyone cares that A's address change is not overtaken by A's earlier one.

The cost is that a partition is the unit of parallelism. Ten partitions means at most ten consumers in a group, and adding an eleventh does nothing. Choosing the count is a one-way door in practice, because increasing it changes which partition a key hashes to, so events for one entity end up split across the old and new partitions and ordering breaks for exactly the keys you were protecting.

Global ordering means one partition, one consumer, and no parallelism. It is occasionally the right answer, for a ledger or an audit log, and it should be a deliberate decision with a throughput number attached rather than something arrived at by accident.

A message that fails is usually retried. Without a limit, a permanently broken message is retried forever: it consumes a consumer slot, it is redelivered ahead of newer work in some brokers, and it can hold up an entire partition while everything behind it waits.

A dead letter queue is where a message goes once it has failed enough times. The main queue keeps flowing, the failure is preserved rather than dropped, and someone can look at it. That last part is the point and the part usually skipped: a dead letter queue nobody reads is a slower way of deleting messages.

Retries need backoff with jitter. Pure exponential backoff keeps every client synchronised, because they all wait the same intervals from the same failure, so the failing dependency is hit by the whole fleet at once, recovers, and is hit again. Full jitter, a random wait between zero and the current ceiling, spreads them out. AWS published the arithmetic on this and the difference is not marginal.

Distinguish what is worth retrying. A 503 or a timeout is transient and deserves the full ladder; a 400, a schema violation or a foreign key that will never exist is permanent, and retrying it eight times over a day just delays the moment someone finds out. Route those to the dead letter immediately, with the error attached, so the queue tells you what is wrong rather than only that something is.

The first distinction is between a queue and a log, and most confusion about brokers comes from missing it. A queue holds work: a message is delivered, acknowledged, and removed, and the queue is empty when the work is done. A log holds a record: messages are appended and kept for a retention period, consumers track their own position, and reading does not consume anything. SQS and RabbitMQ are queues; Kafka is a log.

That difference decides more than it looks. In a log, a new consumer can start from the beginning and rebuild its state, several independent consumers can read the same stream at their own pace, and replaying a bad day is a matter of resetting an offset. In a queue, a message read by one consumer is gone, which is exactly what you want for a unit of work and useless for anything wanting a second opinion on the same events.

SQS is the low-operations option: no cluster to run, effectively unlimited depth, at-least-once with best-effort ordering. Its mechanism worth knowing is the visibility timeout, which hides a message from other consumers while one works on it, defaulting to 30 seconds and extendable to twelve hours. A handler that takes longer than the timeout without extending it will find the same message being processed by somebody else, which is the most common way people discover their consumer is not idempotent. FIFO queues add ordering and deduplication with a throughput ceiling that standard queues do not have.

Kafka is the option that buys retention and fan-out, and charges in operational complexity. Ordering is per partition, consumers commit offsets, and the retention window is a design parameter rather than an implementation detail: it decides how far back you can replay and how long a consumer can be down before it loses data. RabbitMQ sits between them, with routing rules rich enough to express most topologies and per-message acknowledgement, and it is the right answer more often than its unfashionability suggests.

Redis Streams deserve a mention because they are already installed. Consumer groups, acknowledgements and a pending entries list give real queue semantics, at a fraction of the operational cost, with the caveat that durability is Redis durability: an append-only file with a one-second fsync loses up to a second of messages on a hard failure. For work that can be recomputed, that is fine and the simplicity is worth a great deal.

The decision, then, is three questions rather than a comparison table. Does anything need to read these events more than once, or later, which means a log. Does the work need ordering, and within what, which sets the partitioning. And who operates it at 3am, which is the question that most often decides between running a cluster and paying someone else to.