Async messaging & event streaming

Most of what a web service does happens inside a request: a client asks for something, your code runs, and you send a reply. That model is simple and it is correct for the work the caller is actually waiting on. But a surprising amount of what we cram into requests is work the caller does not need to see finish. This lesson is about recognising that second category of work and moving it out of the request path — first the problem it causes, then the tool that fixes it (a message queue), then the realities that bite people who adopt it without understanding them.

The problem: slow, secondary work trapped inside a request

Take a concrete, ordinary feature: a user signs up. To make the account real, several things have to happen. You must write the user row to the database — that one truly is part of "did the signup succeed?". But then you also want to send a welcome email, provision their workspace (create a default project, allocate storage), and index their profile so it shows up in search. The naive version does all of it in the signup request handler, one after another, before returning the response:

def signup(request):
    user = db.create_user(request.email, request.password)
    email_service.send_welcome(user)      # calls an external mail API — ~800ms, sometimes times out
    provisioning.create_workspace(user)   # another service — ~1.2s
    search.index_profile(user)            # yet another — ~400ms
    return Response("Welcome!")            # user has been staring at a spinner for ~2.5s

This is bad in two distinct ways. First, it is slow: the user waits for the sum of every step, even though they only care about "am I signed up?". A 2.5-second spinner for what should be an instant page is a real cost in conversions and perceived quality. Second, and worse, it is fragile: the request now succeeds only if all four systems are healthy at the same instant. If the mail provider is having a slow afternoon, your signup endpoint slows down or times out — even though signing up has nothing to do with email. You have coupled the success of a critical action to the availability of three non-critical ones. If create_workspace throws after the user row is written, do you fail the whole signup? Roll back? Leave a half-created account? Every answer is awkward, and that awkwardness is the symptom of mixing "must happen now" work with "should happen soon" work in one request.

The fix: hand the work off to a broker

The cure is to hand the work off instead of doing it inline. You introduce a piece of infrastructure called a message broker (or message queue) that sits between the code that creates work and the code that performs it. Four terms carry this whole topic, so let us define them precisely:

• Message — a small, self-contained piece of data describing a unit of work or a fact that happened. Usually a short JSON or binary payload, e.g. { "type": "user.signed_up", "user_id": 4821 }. It is not a function call; it is data on a wire.
• Producer (also called a publisher) — the code that creates a message and hands it to the broker, then returns immediately. The producer's job ends the moment the broker has safely accepted the message; it does not wait for the work to be done.
• Broker — the middleware (SQS, RabbitMQ, Kafka, etc.) that receives messages from producers, stores them durably, and delivers them to consumers. It is the buffer and the post office.
• Consumer (also called a worker or subscriber) — a separate process that reads messages from the broker and actually performs the work, on its own schedule. When it finishes a message it tells the broker "done" (an acknowledgement, or ack), and the broker can then drop that message.

The shape is always the same — work flows left to right, and the broker decouples the two ends in time:

  PRODUCER                 BROKER                    CONSUMER
 (signup API)            (the queue)               (worker process)

  create_user
       |
       |  publish "user.signed_up"
       +-------------------->  [ msg ][ msg ][ msg ]  -- delivered later -->  send email
       |                       (stored, waiting)                              provision
  return "Welcome!"                                                           index profile
   (instant)                                                                  ack each one

The rewritten signup handler now does only what the user is waiting on, then publishes one message and returns:

def signup(request):
    user = db.create_user(request.email, request.password)
    broker.publish("user.signed_up", {"user_id": user.id})  # ~2ms, just hands off
    return Response("Welcome!")                                   # total request time ~25ms

The email, provisioning, and indexing now happen in one or more consumer processes that subscribe to that message. The user gets their "you're in" page in milliseconds; the rest happens a moment later, out of sight.

What the hand-off buys you

That one move — handing off instead of waiting — buys four distinct benefits. Each is worth understanding on its own:

• Decoupling — the producer does not know who consumes the message, how many consumers there are, or how long they take. It publishes a fact and walks away. You can add a fifth consumer (say, "send the analytics team a notification") tomorrow without touching the signup code at all. The two sides evolve independently because they share only the message contract, not the code path.
• Buffering / absorbing spikes — the queue acts as a shock absorber. If 50,000 people sign up in the minute after a product launch, those 50,000 messages pile up in the queue and drain at whatever rate the consumers can sustain. Without a queue, that spike would hit your email service and database all at once. Concrete spike example: suppose a consumer can process 500 emails/second but a marketing blast triggers 50,000 signups in a 10-second window (5,000/s). Synchronously, 4,500 requests per second would fail or time out. With a queue, all 50,000 messages are accepted instantly and the consumer works through the backlog in ~100 seconds — every email still gets sent, just slightly later. The queue traded latency for not dropping work.
• Resilience — if a consumer is down, the work is not lost. Messages simply wait in the broker until a consumer comes back, then get processed. Compare to the synchronous version, where a dead mail service meant a failed (lost) signup. Here the signup already succeeded; the welcome email is just delayed until the mail worker recovers. Durability is the broker's job: most brokers persist messages to disk and replicate them, so a broker crash does not vaporise the queue.
• Independent scaling — the email work is slow, but the producer is fast. With a queue you scale only the part that is slow: run ten copies of the email consumer reading from the same queue, and they split the backlog ten ways, with zero changes to the signup service. The producer and each consumer scale separately according to their own load.

Two shapes: work queues vs pub/sub and streaming

There are two fundamentally different delivery shapes, and they answer different questions. Confusing them is the most common mistake in this area, so be deliberate about which one you want.

A few streaming terms you will hear constantly, defined briefly:

• Topic — a named stream of related messages, like a category or channel. Producers publish to a topic (e.g. orders); consumers subscribe to it. It is the unit you address.
• Partition — a topic is split into one or more partitions, each an independent, ordered append-only log. Partitions are how a topic scales: ten partitions can be read by ten consumers in parallel. A message is assigned to a partition, usually by hashing a key (e.g. the user_id), which guarantees all messages with the same key land in the same partition.
• Subscriber / consumer group — the party reading a topic. In streaming systems, a group of consumers cooperates to read a topic, with the partitions divided among the group's members so each partition is handled by exactly one member of that group.
• Offset — a consumer's bookmark: the position (a sequential number) it has read up to within a partition. Because the log is retained, a consumer can commit its offset to remember progress, or rewind the offset to replay earlier events.

The hard realities of production

Queues are not free magic. Four realities trip people up; understanding them up front is the difference between a robust system and a mysterious one.

1. Delivery is usually at-least-once. At-least-once delivery means the broker guarantees a message will be delivered, but it may deliver it more than once. Why? Imagine a consumer reads a message, fully processes it (sends the email), and then crashes before it can send the acknowledgement. The broker never heard "done," so — playing it safe — it re-delivers the message to another consumer. That second consumer sends the email again. The broker chose duplicate-but-never-lost over might-lose, which is almost always the right trade. The consequence is that your consumers must be idempotent: processing the same message twice has the same effect as processing it once. The classic disaster is a non-idempotent payment consumer that charges a customer's card a second time because the "payment requested" message was redelivered after a crash. The fix is to make the operation safe to repeat — for example, record an idempotency key derived from the message ID, and on a duplicate, detect "I already processed message 4821" and skip the charge. (See Distributed systems in practice for idempotency keys, deduplication, and why true exactly-once delivery is mostly a myth.)

2. Ordering guarantees are limited. Beginners assume messages come out in the order they went in. A broker rarely promises that globally. What you usually get is order per partition or per key only: all messages with the same key (say, all events for user 42) arrive in the order they were sent, but two different users' events may interleave arbitrarily. This is a direct consequence of partitioning for parallelism — different partitions are read by different consumers at different speeds, so there is no single global sequence. If your logic depends on global ordering ("apply these account transactions strictly in order across all users"), that is a design constraint you must engineer around (e.g. by keying on the account so each account's events stay ordered within one partition), not something the queue hands you.

3. Backpressure when consumers lag. Backpressure is what happens when producers are putting messages in faster than consumers can take them out: the queue grows. A steadily growing queue depth (or, in streaming, a growing consumer lag — the gap between the latest offset and the consumer's committed offset) is your early-warning signal that consumers cannot keep up. Your two responses are to scale out consumers (add workers to drain faster) or to shed load (reject or sample incoming work). The danger is ignoring it: an unbounded backlog means the work you are doing is arbitrarily stale — emails sent hours late, analytics that lag reality — and may eventually exhaust broker storage. Watching queue depth and consumer lag belongs in your dashboards; see Observability for what to alert on.

4. Poison messages and the dead-letter queue (DLQ). Some messages can never succeed — a malformed payload, a referenced record that was deleted, a bug that throws on one specific input. These are poison messages. If the broker just keeps redelivering a failing message forever, it can block everything behind it and burn resources retrying in a loop. The standard solution is a dead-letter queue: after a message fails N times (say 5 retries), the broker stops redelivering it to the main queue and routes it to a separate side queue — the DLQ — where it sits safely for a human to inspect. Concrete scenario: your image-resize consumer receives a message pointing at a corrupted upload. It throws every time it tries to decode the image. After 5 attempts the broker parks that one message in the DLQ; the consumer moves on to the next image and the queue keeps flowing. Later, an engineer reads the DLQ, sees the corrupt file, fixes the upload validation, and either discards or re-drives the parked message. Nothing was lost, and one bad message never stalled the pipeline.

When to queue vs call synchronously

And queues genuinely cost something — adopt them for the work above, not reflexively. The costs: operational complexity (you now run and monitor a broker, consumer fleets, DLQs, and alerts on lag — more moving parts to keep alive); eventual consistency (right after signup the welcome email has not been sent yet and the profile is not yet in search, so your system is briefly in an in-between state that the UI and your tests must tolerate); and harder debugging (a synchronous bug shows up in one stack trace, but an async failure is split across the producer, the broker, and a consumer running minutes later, so you lean on correlation IDs and tracing to follow one message end-to-end).

Worked example: an order, end to end

Pull it together with an e-commerce checkout, the canonical event-driven example. A customer places an order. The order service does only the must-happen-now work — validate the cart, reserve payment, write the order row — and then publishes one event to the orders topic. Three independent services subscribe to that topic (this is pub/sub: each gets its own copy), and each reacts in its own way, at its own pace, scaling on its own:

                                  +--> [email service]  -- send "Order confirmed" receipt
 customer       order service     |
 places   --->  validate + save   |
 order          payment           +--> [inventory service] -- decrement stock, trigger reorder
                  |               |
                  |  publish      |
                  +--> [ orders ] +--> [analytics service] -- record sale, update dashboards
                       (topic)

 order service returns "Order placed!" to the customer immediately, before any consumer runs.

Trace one order through the realities above. The customer sees "Order placed!" in milliseconds (low latency — the slow downstream work is off the request path). If the analytics service is down for maintenance, the order still succeeds and its event waits in the stream; when analytics restarts it replays from its last offset and catches up (resilience + replay). The email consumer crashes after sending the receipt but before acking, so the broker redelivers — and because the consumer keyed its dedupe on the order ID, it recognises "already emailed order #7781" and does not send a second receipt (at-least-once + idempotency). If the topic is keyed by order_id, all events for one order stay in order within their partition, even though different orders interleave (per-key ordering). And if the inventory service hits a SKU that no longer exists and throws five times, that single event lands in the DLQ for an engineer to inspect, while every other order keeps flowing. One published event, many independent reactions, each robust on its own — that is the whole payoff of async messaging.

In a larger design, queues are how you keep request latency low and services loosely coupled — see System design for where they sit in an end-to-end architecture, and Caching for the other half of keeping requests fast.