Distributed systems in practice
The interview System design lesson is the whiteboard: boxes, arrows, "we'll shard here, cache there." This lesson is the operational reality once those boxes are real machines talking over a real network. A distributed system is simply any program whose parts run on more than one machine (or even more than one process) and coordinate by sending messages. That sounds harmless. It is not. The moment your program crosses a network boundary, a new and nastier category of bug appears — not logic bugs you can reproduce on your laptop, but partial-failure bugs, where one piece is dead or slow and every other piece has no reliable way to find out. Everything below is the kit you reach for to stay correct in the face of that.
The shift: partial failure
On a single machine, failure is wonderfully binary. You call a function; either it returns a value or it throws an exception, and either way you know what happened. If the whole process crashes, everything in it dies together — there is no awkward state where half your program kept running while the other half vanished. Memory you wrote is there when you read it back. A function call costs nanoseconds and never "times out." This total, all-or-nothing clarity is the comfortable world you learned to program in.
Cross a network and that clarity is gone. Now you have two machines, A and B, and the only thing connecting them is a wire that can drop, delay, reorder, or duplicate any message. The defining new phenomenon is partial failure: one part of the system fails while the rest keeps running, completely unaware. Concretely: A sends "charge this card $50" to B. B receives it, charges the card, and sends back "done." But that reply packet is dropped on the way home. From A's point of view, the request simply never came back. A is now stuck with a question it cannot answer from its own information: did B never get the request, or did B do the work and only the reply was lost? On one machine this dilemma does not exist — a function either returned or it didn't. Across a network it is the normal case, and it is the seed of almost every hard problem in this lesson.
In 1994 engineers at Sun Microsystems wrote down a list of false assumptions that newcomers to distributed systems make. Every outage you will ever debug traces back to quietly assuming one of these is true. None of them is:
- The network is reliable. Packets drop, connections reset mid-stream, a switch reboots, a cable gets unplugged by a contractor. Any call across the wire can simply not arrive. Failure it causes: a request you assumed always lands sometimes vanishes, leaving the caller hanging.
- Latency is zero. A local function call is nanoseconds; a network round-trip is milliseconds within a datacenter and tens-to-hundreds of milliseconds across the planet, and it varies wildly. Failure it causes: a loop that makes one remote call per item — fine on a list of 5 in testing — turns into 10,000 sequential round-trips in production and takes minutes.
- Bandwidth is infinite. The pipe has a finite width. Failure it causes: shipping a 50 MB blob per request saturates the link and starves everything else sharing it.
- The network is secure. The path is hostile; anyone on it can read or tamper. Failure it causes: "internal" traffic sent in the clear gets intercepted because someone assumed the perimeter was safe.
- Topology is stable / there is one administrator. Machines come and go, IP addresses change, routes shift, and no single person owns the whole path between A and B. Failure it causes: hard-coded addresses break on the next deploy, and when something is slow, no one team can see end to end.
All of these matter, but one consequence sits underneath the rest — the lost reply. The server did the work; the acknowledgement never made it back; and the caller genuinely cannot tell "it failed" from "it succeeded but I didn't hear." Hold onto that ambiguity. It is the root of CAP, of idempotency, of retries, and of delivery guarantees — every section below is, in some sense, a response to the lost reply.
CAP: the choice a partition forces on you
CAP is a way to reason about what your system does when the network splits. First, three definitions, in plain English:
- Partition (P). The system splits into groups of machines that can still talk among themselves but cannot reach the other group — a network rift cuts your cluster in two. Crucially, partitions are not optional. The network will eventually do this to you, so P is a fact you must plan around, not a feature you can decline.
- Consistency (C). Every read returns the most recent write (or an error). There is one agreed-upon truth and you never see a stale version of it.
- Availability (A). Every request gets a non-error response — the system always answers, even if the answer might be out of date.
The theorem says: during a partition, you cannot have both C and A. Picture two replicas, one on each side of the rift, and a write that lands on the left side. A read now hits the right side, which has not seen that write. You are forced to choose: either the right side answers with its stale value (you kept Availability, you gave up Consistency), or the right side refuses to answer because it can't be sure it's current (you kept Consistency, you gave up Availability). There is no third door while the partition lasts. CAP is not a slogan to recite; it is the decision you are implicitly making every time you design a read path. Two worked examples make the choice concrete:
An account holds $100. During a partition the right side doesn't know whether the left side just approved a $100 withdrawal. If it stays available and answers "$100 available," the customer can spend the same $100 twice — real money lost. So a balance picks consistency: block or reject the read/withdrawal until the partition heals and the truth is known. A spinner beats a double-spend. See Databases for how transactions enforce this.
A post shows 41 likes; someone on the other side of the partition just added the 42nd. If a reader sees 41 for a couple of seconds before it catches up, literally no one is harmed. Here you pick availability: always render a number, accept that it's briefly stale, and let it converge. Refusing to show the page to guarantee an exact count would be absurd.
Consistency models: strong vs eventual
The C/A choice above shows up in everyday API design as a choice between two consistency models — the contract your storage offers about what a read can return.
Every read returns the latest committed write, full stop. Read example: you transfer $100, then immediately refresh — you are guaranteed to see the new balance, never the old one. The cost is latency and availability: the system may have to coordinate across replicas, and during a partition it will block or reject. Use it for money, inventory counts, unique-username checks, anything where a stale answer is a correctness bug.
Reads may be stale for a moment, then all replicas converge to the same value once writes propagate. Read example: you update your profile bio; a friend loading your page in the same second might see the old bio, but a refresh a moment later shows the new one. It is cheaper and stays available under partition. Use it for like counts, view counters, social feeds, caches — places where "right soon" beats "right now, or not at all." See Caching for the canonical eventually-consistent layer.
The fallacies tell you calls will fail and get duplicated; the consistency choice tells you what truth you're willing to serve. The patterns below are how you build something correct on top of all of that. Internalize them — they recur in every networked service you will ever touch.
An operation is idempotent if performing it twice (or ten times) has exactly the same effect as performing it once. Reading a value is naturally idempotent; "set balance to $100" is idempotent; but "add $50" and "charge the card" are not — each repeat changes the result again. Idempotency is the property that makes retries safe, and retries are unavoidable, because of the lost reply: the caller times out, cannot tell success from failure, and therefore must retry to be safe. If the operation is idempotent, that retry is harmless even when the original actually succeeded.
The standard technique for making a naturally non-idempotent operation idempotent is the idempotency key. The client generates a unique id for the intent ("this one specific charge") and sends it with the request. The server records, atomically, "I have processed key X, and here is the result." When the same key arrives again — a retry of the same intent — the server skips the work and returns the stored result instead of charging the card a second time. The key turns "did this already happen?" from an unanswerable network question into a cheap lookup the server can answer locally.
# Naive: the lost reply makes the caller retry,
# and a second call charges the customer AGAIN.
def pay(card):
return charge(card, 50) # retry == double charge
# Idempotent: keyed on the client's unique id.
# A retry with the same key is a no-op that
# returns the FIRST result, so the card is charged once.
def pay(card, key):
if key in processed: # seen this exact intent before?
return processed[key] # return stored result, do NOT re-charge
result = charge(card, 50)
processed[key] = result # remember it (must be atomic + durable)
return resultIn a real system processed is a durable, shared store (a row with a unique constraint on the key, or a Redis SETNX), so the dedup survives a server restart and works across many server instances — not the in-memory dict shown here.
Timeouts, retries, backoff, and jitter
Given that calls fail and replies get lost, you need a disciplined recipe for calling another service. Each ingredient fixes a specific failure mode, and skipping any one of them reintroduces an outage.
- Always set a timeout. A timeout is the maximum time you'll wait for a reply before giving up. A call with no timeout does not "fail" when the dependency dies — it hangs forever, pinning the thread that made it. Now imagine every incoming request triggers one such hung call: thread by thread your service runs out of workers and wedges completely, all because one downstream dependency went silent. The timeout converts an infinite hang into a clean, handleable error.
- Retry — but bounded. Because a single failure is often a transient blip (a dropped packet, a momentary GC pause), retrying a few times recovers gracefully. But retry a fixed, small number of times — an unbounded retry loop against a truly-down service just hammers it forever.
- Use exponential backoff. Don't retry instantly. Wait 1s, then 2s, then 4s, doubling each time. Instant retries against a service that is already struggling pile on more load at exactly the worst moment and keep it down. Backing off gives it room to recover.
- Add jitter. Here is the subtle trap: if 10,000 clients all hit the same failure at the same instant and all back off by exactly 2s, they retry in perfect lockstep, slamming the recovering server in synchronized waves — the thundering herd (a.k.a. retry storm). Each wave knocks it back down, triggering the next wave. The fix is jitter: randomize each client's wait so the retries smear out over the window instead of arriving all at once. Spreading the herd is the difference between a smooth recovery and a self-inflicted permanent outage.
import random, time
def call_with_retries(max_retries=5):
for attempt in range(max_retries):
try:
return call(timeout=2) # 1) bounded wait, never hang
except Timeout:
if attempt == max_retries - 1: # 2) bounded retries
raise
backoff = 2 ** attempt # 3) 1s, 2s, 4s, ...
time.sleep(random.uniform(0, backoff)) # 4) + jitterNote the deep link to the previous section: retries are only safe if the operation is idempotent. Backoff and jitter decide when to retry; idempotency decides whether retrying is even allowed. You need both.
Circuit breakers: fail fast to stop cascades
Retries handle a blip, but if a dependency is genuinely down, retrying just burns your own resources and prolongs everyone's pain. A circuit breaker is a wrapper around a remote call that watches the recent failure rate and, when failures cross a threshold, stops even trying for a while. Borrowing from electrical breakers, it has three states:
- Closed — normal operation. Calls flow through; the breaker counts failures.
- Open — tripped. Too many recent failures, so the breaker fails fast: it immediately returns an error or a fallback without making the doomed network call at all, for a cooldown window.
- Half-open — probing. After the cooldown, it lets a single trial request through. If that succeeds, it assumes the dependency recovered and flips back to closed; if it fails, it snaps back to open for another cooldown.
Concrete scenario. Your checkout service calls a recommendations service to show "you might also like." Recommendations crashes. Without a breaker, every checkout request waits the full 2s timeout for recommendations, threads back up, and the checkout page — which should still work without recommendations — grinds to a halt, taking down sales. With a breaker, after a handful of failures it opens, instantly returns "no recommendations" (a graceful fallback), checkout stays fast, and once recommendations recovers the half-open probe quietly closes the circuit again. Failing fast on the sick dependency is what prevents one service's outage from cascading into a system-wide collapse.
Delivery guarantees, and why "exactly-once" is a myth
When you send a message over an unreliable network, the lost reply leaves you exactly two honest delivery guarantees, and they are opposites:
- At-most-once. Send it and never retry. If it's lost, it's lost. You will never process a duplicate, but you may drop messages. Acceptable for things like a best-effort metrics ping where losing one sample doesn't matter.
- At-least-once. Retry until you get an acknowledgement. You will never lose a message, but because the ack can be the thing that got lost, you may process some messages more than once (duplicates). This is the safe default for anything you can't afford to drop.
What about exactly-once — delivered once, never lost, never duplicated? Over an unreliable network, true end-to-end exactly-once delivery is essentially impossible: the sender can never be certain whether to retry, so it must either risk loss (at-most-once) or risk duplication (at-least-once). There is no magic third option on the wire. What people call "exactly-once" in practice is at-least-once delivery plus idempotent processing on the receiving end. Duplicates still physically arrive; you simply make the handler not care, so the observable effect happens once. That's the whole recipe, and it ties the lesson together: at-least-once + idempotency = effectively-once.
A message queue (Kafka, SQS, RabbitMQ) sits between the service that produces work and the one that consumes it. The producer drops a message and immediately moves on; the consumer pulls messages when it has capacity. This decoupling buys you three things: a buffer that absorbs traffic spikes instead of overwhelming the consumer, isolation so that if the consumer is temporarily down the messages wait safely in the queue rather than being lost, and the freedom to scale each side independently. The catch you already know: nearly all queues deliver at-least-once, so the same message can be handed to your consumer twice — which means your consumers must be idempotent. Same lesson, one more time. For the full treatment, see Message queues.