The operational layer: cost, observability, rollback
A design that works on the whiteboard and a design you'd actually be paged for at 3am are different designs. At mid and senior level, interviewers now explicitly grade whether you think about what happens after launch: what it costs to run, how you'd know it's broken, and how you'd undo a bad change. These used to be bonus points. They're now table stakes — skipping them leaves obvious senior signal on the table.
📖 Walk me through it — plain English
Building a system is like opening a restaurant. Drawing the kitchen layout is the easy part. The thing that separates someone who has actually run a kitchen from someone who's only drawn one is the three questions they ask without being prompted: What does it cost me to keep the lights on? How do I know when a dish is going out wrong before customers complain? And when I change the menu and it bombs, how fast can I put the old menu back?
In systems those three are cost, observability, and rollback. Cost means knowing roughly what your design spends — the database, the cache, the bandwidth — and whether a cheaper shape would do. Observability means you can ask new questions of a running system: not just "is it up?" but "why are users in Tokyo seeing slow checkouts since 2pm?" You get that from three data streams — logs (what happened), metrics (numbers over time), and traces (the path one request took through your services). Rollback means every change ships in a way you can reverse fast — behind a flag, to a slice of traffic first, with a one-click path back to the last good version.
You don't need to recite cloud pricing. You need to volunteer these three concerns in the back half of the round, the way a senior naturally would, and tie each to your specific design.
Cost: the back-of-envelope you should reach for
You won't be quizzed on exact prices, but you should be able to spot which part of your design dominates the bill and name a cheaper alternative. The usual order of expense:
- Egress bandwidth (data leaving the cloud to users) is often the sneaky-biggest line item — which is exactly why a CDN pays for itself: it serves bytes from the edge and offloads your origin.
- Storage is cheap per GB but adds up at scale and over time (retention). Naming a TTL or a tiering plan — hot data on SSD, cold data on cheap object storage — is a senior move.
- Compute scales with traffic; the lever is efficiency (a cache that turns 1000 DB reads into 1) and right-sizing, not just "add servers."
- Cross-region replication and chatty service-to-service calls quietly multiply both latency and bill.
Traced example. Design serves 100M image reads/day at ~200KB each. Straight from your servers that's 20 TB/day of egress — the dominant cost. Put a CDN in front: now ~95% of those bytes are served from the edge, your origin egress drops ~20×, and your servers stop spending CPU on static bytes. One sentence — "I'd front images with a CDN, mostly to cut egress cost, not just latency" — shows you cost the design, not just drew it.
Observability: the three pillars
"Observability" is the ability to ask new questions of a running system without shipping new code to answer them. It stands on three data streams — define each on first use:
- Logs: timestamped records of discrete events ("user 42 checkout failed: card declined"). Great for the specific story of one request; expensive to search at scale.
- Metrics: numbers aggregated over time (requests/sec, error rate, p99 latency). Cheap to store and alert on; tell you that something's wrong and how much.
- Traces: the end-to-end path of a single request as it hops across services, with timing at each hop. Tell you where the time or the error went in a distributed call chain.
The number that matters most is tail latency, usually p99 — the response time your slowest 1% of requests see. Averages lie: a system with a great average can still have 1% of users timing out. Watching p99 (and p999) is the difference between "looks fine on the dashboard" and "actually fine for users." Pair metrics with SLI/SLO/error-budget thinking: an SLI is a measured number (say, % of requests under 300ms), an SLO is its target (99.9%), and the error budget is the allowed miss (the leftover 0.1%) — which tells you when to stop shipping features and go fix reliability.
Rollback: every change must be reversible
Senior engineers assume a fraction of changes will be bad, so they make every change easy to undo fast. The vocabulary:
- Feature flag: a runtime toggle that ships code "dark" and turns it on (or off) without a redeploy. Rollback becomes flipping a switch in seconds, not reverting and re-deploying for 20 minutes.
- Canary deploy: release to a tiny slice of traffic (1%), watch the metrics, then ramp. If the canary's error rate spikes, you've hurt 1% of users, not 100%.
- Blue-green: run two identical environments; flip all traffic from old (blue) to new (green) at once, and flip back instantly if it's bad.
- Backward-compatible migrations: change schemas in steps (add column → backfill → start writing → stop reading old) so the new and old code can both run during the rollout — otherwise you can't roll back without data loss.
The graceful-degradation reflex. "What happens when dependency X is down?" should have an answer for every external call in your diagram. A good one: serve stale cache, queue the write for later, show a reduced experience — anything but a hard 500. Naming the fallback for your slowest/riskiest dependency is pure senior signal.
Pitfalls
Optimizing mean latency while p99 quietly times out. Always reason about the tail — that's the experience your unhappiest users actually have.
Observability is a design input, not an afterthought. If you can't see it, you can't operate it. Name the key metrics you'd alert on while you design.
A schema change that the old code can't read locks you into the new version. Stage migrations so rollback stays possible the whole way.
Every external dependency can be down. A design with no degradation plan is a design that turns one outage into a total outage.
Takeaway: in the back half of a design round, volunteer the operational layer. Cost: name which part of your design dominates the bill (often egress) and the cheaper shape. Observability: logs, metrics, traces — and watch p99, not the average, against an SLO/error budget. Rollback: flags, canaries, blue-green, and backward-compatible migrations so every change is reversible, with a graceful-degradation answer for each dependency. Naming these unprompted is one of the clearest mid-to-senior signals you can give.
→ Going deeper: apply the cost lens to scaling primitives (caches and CDNs are cost levers, not just speed) and bring the rollback vocabulary into rate limiter and the case studies that follow. Use the CBW habit from defending tradeoffs when you state each operational choice.