Lesson 46 · System design

Design a chat system

📖 Walk me through it — plain English

This is a system design question: the interviewer hands you a big, vague goal ("build a chat app like WhatsApp or Slack") and watches how you break it into sensible pieces. There is no single right answer and no code to run — they are testing whether you can reason about trade-offs out loud. The whole lesson is structured the way a good answer flows: first you clarify what you're actually building, then you make a few key decisions (how messages travel, how data is stored), then you call out the hard parts.

The single biggest decision is the delivery protocol — how a message gets from one phone to another in real time. Normal web pages work by "request then response": your browser asks the server for something, gets an answer, and the line goes quiet. That's fine for loading a page, but chat needs the server to push a message to you the instant someone sends it, without you constantly asking "anything new? anything new?". The lesson lists three options and lands on WebSocket: a single connection that stays open and lets data flow both directions (you send, and the server pushes to you) over that same pipe. The other two are weaker — long polling keeps re-asking and wastes resources at scale, and Server-Sent Events only push one way (server → you), so they can't carry your outgoing messages.

The analogy: think of a normal web request like mailing a letter and waiting for a reply — slow, one round-trip at a time. WebSocket is like keeping a phone line open between you and the server: nobody hangs up, and either side can talk the moment they have something to say. That open line is what makes typing indicators and instant message pops possible.

Now the tricky part the lesson centers on: connection routing. At scale you don't have one server — you have hundreds ("chat server, one of many"). Your open phone line lands on server Y; your friend's open line lands on a different server X. So when you send "hi", server Y has no direct way to reach your friend, because it isn't holding your friend's connection. The fix is a message bus (a shared relay like Kafka or Redis Pub/Sub — "pub/sub" just means publish a message and any interested server can subscribe to receive it). Y publishes the message to the bus; X is subscribed, grabs it, and pushes it down the line it's holding to your friend.

Step 1 · You (Alice) send "hi". It travels up your open WebSocket to whichever chat server holds your line — server Y. Alice's cell is highlighted as the active sender.

Alice → Y

bus

Bob

Step 2 · Server Y doesn't hold Bob's connection, so it can't push directly. Instead it publishes the message onto the shared message bus (Kafka / Redis Pub-Sub).

Alice → Y

bus

Bob

Step 3 · Server X is subscribed to the bus and holds Bob's open WebSocket. It picks up the message off the bus.

Alice → Y

bus

Bob

Step 4 · X pushes "hi" down Bob's WebSocket — message delivered. (In parallel, a persistence service writes it to the message store so history survives; if Bob were offline, a push-notification service would alert him instead.)

Alice → Y

bus

Bob ✓

Two more "hard parts" worth understanding plainly. Ordering: messages can arrive out of sequence, so each message gets a Snowflake ID — an ID with a timestamp baked into the front, so sorting by ID also sorts by time — and writes for one conversation are handled one-at-a-time so they can't scramble. Idempotency (a fancy word for "doing it twice has the same effect as doing it once"): phones on flaky networks retry, so a single "hi" might get sent twice. The client attaches a client_msg_id tag; if the server sees that same tag again, it knows it's a duplicate and drops it, so Bob doesn't see "hi" twice. For group chats, the same bus does "fan-out" — one published message gets copied to every member's server.

Why this design holds up: because no single server has to know where everyone is. Each server only manages its own open connections and talks to the shared bus. That lets you add more chat servers as users grow without rewiring everything — the bus is the universal meeting point, and persistent storage (something like Cassandra or a sharded SQL database, "sharded" meaning the data is split across many machines by conversation) keeps the full history safe and searchable.

The vocabulary, defined once

Before the design, here is every term this lesson leans on, each in one plain sentence. Skim it now; refer back when a word reappears.

Real-time delivery — the server pushes a message to the recipient within a fraction of a second of it being sent, instead of the recipient having to ask for it.
WebSocket — a connection that opens once and stays open, carrying data in both directions (client→server and server→client) over the same pipe. The right default for chat.
Long polling — the client makes an ordinary HTTP request; the server holds it open (doesn't answer yet) until there is news, then replies, and the client immediately asks again. Simple, but one held request per user is wasteful at scale.
SSE (Server-Sent Events) — a long-lived HTTP stream that pushes data one way only, server→client. Good for live feeds/notifications; can't carry the user's outgoing messages, so it's a poor fit for two-way chat.
Message queue / message bus / pub-sub — a shared relay (Kafka, Redis Pub/Sub, NATS) where one service publishes a message and any other service that subscribed to that channel receives a copy. It decouples senders from the servers holding recipients' connections.
Fan-out — taking one incoming message and copying it out to many destinations (e.g. every member of a group, or every device a user has).
Online presence — knowing whether a user is online/offline/typing right now, usually tracked as a short-lived key that the client keeps refreshing.
Delivery receipt / read receipt — a small status update flowing back to the sender: "the message reached Bob's device" (delivered) and later "Bob opened it" (read).
Message ordering — the guarantee that messages in a conversation appear in a sensible, agreed sequence rather than scrambled.
Idempotency — a property where doing an operation twice has the same effect as doing it once; here, it stops a retried send from showing up as a duplicate.
At-least-once delivery — the system promises a message is delivered one or more times (never zero), which means duplicates are possible — which is exactly why idempotency matters.
Push notification — an alert delivered through Apple/Google's notification services (APNs/FCM) to a phone that has the app closed or is offline.
Sharding — splitting one logical dataset across many machines by some key (here, by conversation), so no single machine holds everything.
End-to-end encryption (E2EE) — encrypting the message on the sender's device so only the recipient's device can decrypt it; the servers in between relay ciphertext they can't read.

▶ Clarify

1:1 only, or also group chats? Read receipts? Presence (online/offline/typing)?
Message ordering: per-conversation? Globally?
Offline delivery + history retention?
Scale: DAU, peak concurrent connections, messages/sec.

Why each question earns its place: every answer eliminates whole branches of the design, so asking them first is not stalling — it is scoping.

1:1 vs group, receipts, presence — these are the functional requirements (what the system must do). Groups force you to handle fan-out; read receipts add a write on every "seen"; presence adds a constantly-churning piece of state. If they're out of scope, you can simplify.
Ordering scope — guaranteeing order within one conversation (per-conversation) is achievable and is what users actually perceive. A global total order across all conversations is far harder and almost never needed; flagging that distinction shows judgment.
Offline delivery + retention — decides whether you need a durable message store and a separate push-notification path. "Disappearing messages" vs "full searchable history forever" are very different storage bills.
Scale numbers — these are the non-functional requirements. DAU (daily active users), peak concurrent connections (how many WebSockets are open at once — this sizes your connection tier), and messages/sec (this sizes the bus and the write path). A back-of-envelope estimate here justifies every later choice.

▶ Delivery protocol — pick one

Long polling: simplest. Client opens HTTP, server holds open until data, returns, client re-opens. Wastes connections at scale.
Server-Sent Events (SSE): one-way push; great for read-mostly feeds, not for chat (needs upstream).
WebSocket: bidirectional, persistent. Standard for chat. Each connection lives on one chat server.

The whole choice comes down to one question: who initiates the message, and how often does the line need to carry traffic both ways?

Long polling works everywhere (it's just HTTP) and is the easy fallback, but every waiting user ties up a request the whole time, and there's a tiny gap between one poll closing and the next opening where a message can be delayed. Fine for small scale or as a compatibility fallback.
SSE fixes the "keep re-asking" waste with a single long-lived stream — but it only flows server→client. Chat needs the client to send too, so you'd be bolting separate HTTP POSTs on top for outgoing messages. Workable but awkward; SSE shines for notifications and live dashboards.
WebSocket pays the connection handshake once, then keeps a full-duplex (both-directions-at-once) pipe open. Sends, receipts, typing indicators, and incoming messages all ride the same socket. This is why it's the industry default for chat.

The catch to say out loud: a persistent connection is stateful — it lives on exactly one chat server. That single fact is what creates the connection-routing problem we solve later with the bus, and it's why reconnect handling (below) matters.

The API surface

A chat system is two APIs glued together: a small WebSocket protocol for the live message flow, and a few ordinary HTTP/REST endpoints for everything that doesn't need a live pipe (history, conversation management). Spelling this out early frames the whole design.

Over the WebSocket (live)

send — client→server: {client_msg_id, conversation_id, body}. The client_msg_id is the idempotency key.
ack — server→client: "received, here is the server msg_id and timestamp."
message — server→client: a new message pushed into a conversation you're in.
receipt — both ways: delivered / read status updates.
typing / presence — lightweight, fire-and-forget status pings.

Over HTTP/REST (request-response)

GET /conversations — list a user's conversations.
GET /conversations/{id}/messages?before={msg_id} — paginate history (scroll up).
POST /conversations — create a 1:1 or group conversation.
POST /conversations/{id}/read — advance the read marker.
GET /ws-token — get a short-lived token, then dial the WebSocket with it.

The split is deliberate: the live socket carries the hot path (new messages, status) where latency must be tiny; REST carries the cold path (fetching old history) where ordinary caching and pagination are fine. Loading a conversation = one REST call for the last page of history, then the socket streams anything new from that point on.

▶ Architecture

client ←──WebSocket──→ chat server (one of many)
                              ↓ publish
                       message queue / pub-sub (Kafka/Redis Pub-Sub)
                              ↓
                       chat server holding recipient's WS → push to recipient
                              ↓
                       persistence service → write to message store (Cassandra/sharded SQL)
                              ↓
                       offline / push-notification service if recipient is disconnected

Read top to bottom, each box has one job:

Chat servers (the connection tier) — each one holds a pile of open WebSockets and does nothing clever about routing. It only knows the connections it personally holds. A load balancer spreads new connections across them; you scale this tier purely by the count of concurrent connections.
Message bus — the universal meeting point. A sending server publishes here; the bus is what lets a message reach a recipient whose socket lives on a totally different server. It also gives you durability and replay if a consumer falls behind.
Persistence service — subscribes to the same stream and writes every message to the durable store, so history survives restarts and lets users scroll back. Writing is separated from delivering so a slow disk never delays the live push.
Push-notification service — also watches the stream; when the recipient has no live connection, it routes the message to APNs/FCM so the phone buzzes instead.

The key architectural idea: delivering, storing, and notifying are three independent subscribers to the same published message. One publish, three jobs, none blocking the others.

The message flow, step by step

Now walk a message end to end. The diagram in the plain-English section showed the happy path for 1:1; here is the same flow written out, plus the group case.

1:1 message (Alice → Bob)

Alice's client sends {client_msg_id, conversation_id, body} up her WebSocket to server Y.
Y assigns a Snowflake msg_id (timestamp-prefixed, so it sorts by time), dedupes on client_msg_id (idempotency — a retry of the same send is dropped here), and acks Alice with the assigned id.
Y publishes the message to the bus, keyed so it lands on the partition for this conversation (this is what preserves per-conversation order — see below).
Three subscribers act in parallel: the persistence service writes it to the store; the delivery path looks up where Bob's connection lives (presence:bob → server_X) and hands the message to X, which pushes it down Bob's socket; the push service fires a notification only if Bob has no live socket.
When Bob's device shows the message it emits a delivered receipt; when Bob opens the chat, a read receipt flows back to Alice the same way (publish → route → push).

Group message (fan-out)

Identical up to the publish. The difference is the recipient set: the conversation has many member_ids, so delivery must reach each member.
Fan-out happens at the delivery layer: for each online member, look up which server holds their socket and push there; for each offline member, queue a push notification.
For small groups this is trivial. For large groups (think a 5,000-member channel), naive fan-out to every member on every message is expensive — a common refinement is to fan out only to currently online members and let offline members pull history on reconnect, and to batch notifications.

▶ Data model

conversations(id, type, member_ids[])
messages(conversation_id, msg_id (snowflake), sender_id, body, ts) — sharded by conversation_id.
Read state: last_read(user_id, conversation_id, msg_id).
Presence: short TTL key in Redis (presence:user_id → server_id).

Why each table looks the way it does:

conversations — type distinguishes 1:1 from group; member_ids[] is the recipient list fan-out reads. Small and read-heavy, so it caches well.
messages, sharded by conversation_id — the access pattern is always "give me the recent messages in this conversation," so making conversation_id the shard key (the column that decides which machine the row lives on) keeps a whole conversation's history co-located and fast to scan. Within a shard, the Snowflake msg_id doubles as the sort key, so reading in order is free. A wide-column store like Cassandra fits this "huge append-only log, read by recent-first" pattern; a sharded SQL setup works too.
last_read — one row per (user, conversation) holding the highest msg_id they've seen. "Unread count" = messages newer than that marker; read receipts = this value flowing back to the sender. Cheap to update, no need to mark each message individually.
presence — a short-TTL (time-to-live: it auto-expires) Redis key mapping a user to the server holding their socket. The TTL is the trick: if a server dies, its presence keys simply expire instead of lingering as stale "online" state. The client refreshes it with a periodic heartbeat.

Scaling: where the load goes

Each part of the system grows along a different axis, which is exactly why they're separate tiers. Naming the bottleneck for each is what an interviewer wants to hear.

Connection tier

Bottleneck = open sockets, not throughput. A box can hold a fixed number of WebSockets (memory per connection). Scale = add more chat servers behind the load balancer. State is the danger — a downed server drops every socket it held, so clients must reconnect cleanly.

Message bus

Bottleneck = messages/sec. Partition by conversation_id so load spreads while order within a conversation is preserved. Durability here gives you replay if a consumer lags.

Message store

Bottleneck = write volume + total size. Sharding by conversation spreads writes; messages are append-only and rarely updated, which suits log-structured stores. Old history can move to cheaper cold storage.

Presence / fan-out

Bottleneck = churn and large groups. Presence updates are high-frequency but tiny and disposable (Redis + TTL). Large-group fan-out is the real cost — favor fan-out-to-online + pull-on-reconnect.

▶ Tricky bits

Connection routing: recipient's WS lives on chat server X. Sender lives on server Y. Y publishes via the bus; X subscribes and pushes.
Message ordering: per-conversation, use Snowflake IDs (timestamp-prefixed) and serialize writes per conversation.
Idempotency: client sends client_msg_id; server dedupes on retry.
Group chats: fan-out at the queue layer.

Each "tricky bit" expanded, plus the failure it prevents:

Connection routing — restated: because a socket is stateful and lives on one server, no server can reach a recipient directly. The bus is the indirection. The presence map (presence:user → server) is how the delivery path finds the right server to hand the message to.
Message ordering — two ingredients together: order-bearing Snowflake IDs (the timestamp prefix gives a sortable order) and routing a conversation's messages through a single partition so writes for that conversation are serialized (handled one-at-a-time). Either alone is not enough; together they give a clean total order within a conversation, which is what users perceive.
Idempotency & at-least-once — the bus and clients both retry on uncertainty, so the system is at-least-once (a message may arrive more than once). The client_msg_id dedupe key converts that into effectively-once: the server remembers ids it has seen and drops repeats, so Bob never sees "hi" twice.
Group fan-out — one publish, copied to every member. Cheap for small groups; for large ones, fan out only to online members and let the rest pull on reconnect, and batch the push notifications.

Key tradeoffs to say out loud

Push vs pull

Push (server delivers the instant a message arrives) gives the lowest latency and the real-time feel, but it's stateful and costs a held connection per user. Pull (client periodically asks "anything new?") is stateless and dead simple, but it's laggy and wasteful at scale. Real systems are push for the live path, pull for history and for catching up after a reconnect. Large-group fan-out leans pull on purpose to avoid pushing to millions.

Ordering guarantees: how strong?

Per-conversation order is achievable and is what users actually notice, so it's the standard target. Global total order (one sequence across every conversation) would force everything through a single choke point — enormous cost for a guarantee nobody can perceive. Stating that you'll guarantee order within a conversation and explicitly not globally is a sign of good judgment.

Pitfalls

Lost messages. If you push to the live socket but skip durable persistence, a recipient who's briefly disconnected loses the message forever. Fix: persist before (or in parallel with) delivering, and have clients fetch any gap on reconnect using the last msg_id they hold.
Out-of-order arrival. Concurrent publishes or parallel consumers can scramble a conversation. Fix: order-bearing Snowflake IDs + a single partition per conversation so writes are serialized; clients can also sort by msg_id defensively.
Duplicates. At-least-once retries mean the same send can land twice. Fix: dedupe on client_msg_id at the server, and have clients ignore a msg_id they already display.
Thundering herd on reconnect. If a chat server (or a whole zone) drops, every client it held reconnects at once, hammering the load balancer and presence store. Fix: clients reconnect with randomized exponential backoff with jitter (wait a growing, slightly-random delay before retrying) so the surge spreads out instead of arriving as one spike.
Stale presence. If "online" state doesn't expire, a crashed server leaves users showing online forever and messages routed to a dead socket. Fix: short-TTL presence keys refreshed by heartbeat, so failure self-heals by expiry.

A note on end-to-end encryption

Apps like WhatsApp and Signal add end-to-end encryption (E2EE): the message is encrypted on the sender's device with a key only the recipient's device holds, so every server in between — bus, chat servers, store — relays ciphertext it cannot read. The architecture above is unchanged (you're still routing and persisting opaque blobs), but two things shift: the server can no longer do content-based features (search, server-side spam filtering, link previews) because it can't see the text, and you now need a key-exchange / device-management layer so each user's devices can establish shared keys. For an interview, it's enough to say "E2EE means the servers move blobs they can't read; it changes what the server can do with content, not how messages are routed." Mention it as an extension unless the prompt asks for it.

Go deeper (optional): if you want the canonical version of this question, see the "Design a Chat System" chapter in System Design Interview, Vol. 1 (Alex Xu), and the engineering write-ups behind real systems — WhatsApp's use of Erlang for millions of connections per box, Slack's real-time messaging architecture, and Signal's protocol docs for the E2EE details. None are required to answer well; the reasoning above is self-contained.

→ Going deeper: Chat needs partitioning and presence at scale. See Scaling primitives.