You own it in production — so how do you know it's healthy?
Writing code that passes its tests on your laptop is only half the job. The other half is knowing it works right now, in production, for real users — and being able to find out why when it doesn't. The hard part is that you cannot stop production and step through it line by line. There is no debugger attached to a live system serving thousands of people per second; pausing it to inspect a variable would freeze the whole service. So you need a different skill: the ability to ask questions of a running system from the outside, using the trail of data it emits as it runs. The day your service pages you at 2am, the difference between a 5-minute fix and a 3-hour outage is whether you built that ability in advance.
This lesson covers two related things. Observability is the technical capability — the logs, metrics, and traces that let you see inside a black box you can't pause. Site Reliability Engineering (SRE) is the set of practices Google popularized for running services with that data: how to measure reliability as a number, how to alert without going mad, and how to respond to incidents so that "the site is down" becomes a calm checklist instead of panic. Every term below is defined the first time it appears, so you can learn the whole topic from this page.
The three pillars: logs, metrics, traces
Almost everything you can know about a running system comes from one of three kinds of data. They are called the three pillars of observability. Each answers a different question, and a mature system emits all three because no single one is enough.
- Logs — discrete events written out as the code runs, each a timestamped line of detail.
2026-06-03T14:02:11Z user=42 action=checkout result=DECLINED reason=insufficient_funds. Logs are unbeatable for "what exactly happened to this one request?" — they carry the messy specifics. They are cheap to write but expensive to search at scale: finding one line among billions per day is slow and storage-heavy. Prefer structured logs (key=value or JSON) over free-form prose, because you can filter and aggregate structured fields. - Metrics — numbers measured over time and cheaply aggregated: requests per second, error rate, p99 latency, CPU percentage, queue depth. A metric is a time series — the same number sampled again and again — so it's perfect for dashboards (a graph of error rate over the last hour) and for alerts (fire when error rate crosses 1%). Their weakness is the flip side of their strength: a metric is a summary, so it tells you that something is wrong ("errors spiked at 14:00") but rarely why ("which user, which endpoint, which input"). Metrics are pre-aggregated, so they're cheap to store for a long time even at high traffic.
- Traces — the end-to-end path of one request as it travels across many services. In a system split into microservices, a single user action fans out into calls to several backends; a trace stitches those calls into one timeline. Each segment (the time spent inside one service) is a span; the spans share a trace ID so they can be linked. A trace shows that one checkout spent 8ms in the API gateway, 12ms in auth, and 900ms waiting on the payment service. This is the only way to find which hop is slow when the slowness lives across a service boundary.
Concrete "slow checkout" scenario, and which pillar finds what. A user complains that checkout is slow. Your metrics dashboard shows p99 checkout latency jumped from 300ms to 3s at 14:00 — that's the alarm: it tells you there's a real, widespread problem and roughly when it started, but not why. You open a trace for one slow checkout and see 2.9 of the 3 seconds were spent in a single span calling the payment service — now you know where. You search the payment service's logs for that trace ID and find connection pool exhausted, waiting for free connection — now you know why. Metrics found the problem, traces localized it, logs explained it. Each pillar handed off to the next.
Monitoring vs observability
The four golden signals
Metrics can be endless, so SRE prescribes a starting set: the four golden signals. If you can watch only four numbers for any user-facing service, watch these. Each warns about a different category of trouble.
The average latency lies, because a few very slow requests get hidden among many fast ones. Percentiles fix this. The p50 (median) is the value that half of requests are faster than. The p99 is the value that 99% of requests are faster than — equivalently, the slowest 1% are slower than this. So "p99 = 2s" means one in every hundred requests takes longer than two seconds. That tail matters enormously: if a user loads a page that makes 100 backend calls, they'll almost certainly hit at least one p99-slow call, so your p99 is a typical user's experience, not a rare edge case. A service can have a lovely 50ms average and still be painful because its p99 is 5s. Always watch the tail (p95/p99/p99.9), not the mean.
SLI, SLO, error budget — in depth
These three turn "reliability" from a vibe ("the site feels flaky lately") into a number you can manage and argue about with data. They build on each other.
100% − SLO. For a 99.9% SLO the budget is 0.1% of events. This is not failure you tolerate by accident — it is failure you are explicitly allowed to spend.(An aside: an SLA — Service Level Agreement — is the contractual version you promise to customers, with penalties (refunds) if you miss it. Your internal SLO is normally set tighter than the SLA, so you notice trouble and react before you breach the contract.)
Worked numeric example. Take a 99.9% SLO ("three nines") measured monthly. The error budget is 0.1% of the month.
- As downtime: a 30-day month is about 43,200 minutes. 0.1% of that is roughly 43 minutes of allowable badness for the whole month. (For contrast, 99.99% — "four nines" — leaves only ~4.3 minutes; 99% leaves a generous ~7.2 hours. Each extra nine costs ~10× the engineering effort.)
- As requests: if you serve 100 million requests this month, 0.1% is 100,000 requests that are allowed to fail before you've blown the budget.
The clever part is the error-budget policy: a rule, agreed in advance, for what happens as the budget is consumed (this is called the burn). 100% reliability is impossible and not worth chasing, so while there is budget left, the team has permission to ship fast and take risks — that's what the budget is for. But the policy flips when the budget is spent. A concrete decision rule:
If the rolling-30-day error budget is exhausted (or burning so fast it will be gone before the window resets), then a feature freeze takes effect automatically: no risky launches, no non-essential deploys, and the team's effort shifts to reliability work (fixing the top causes of the burn) until the budget recovers. While there is healthy budget remaining, ship freely.
That single rule ends the eternal "ship features vs. stay stable" argument between product and engineering. Nobody has to win on willpower or seniority; the budget number decides. Out of budget means stability mode; in budget means feature mode. The fight becomes a dashboard reading.
Alerting: page on symptoms, not causes
An alert that pages someone (sends a phone-buzzing, wake-you-up notification) should mean one thing: a human needs to act now. The golden rule is to page on symptoms the user actually feels — "error rate breached the SLO," "checkout latency is over 4s," "traffic dropped to zero" — and not on internal causes like "CPU at 85%." High CPU might be completely fine; maybe the service is busy and perfectly healthy. A page tied to a cause fires when nothing is actually wrong for users.
Why this matters so much: bad alerts cause alert fatigue. Concretely, suppose you set a page on "CPU > 80%." It fires every afternoon at peak traffic even though latency and errors are perfectly normal. After two weeks of being woken for nothing, the on-call engineer starts ignoring or muting that alert — and then the one night CPU spikes because a real bug is melting the service, the page gets swiped away with all the others. When everything pages, people mute everything, and the real fire is the one that gets ignored. Good alerting is therefore deliberately sparse: few alerts, each one actionable, each tied to real user pain. Causes belong on dashboards and runbooks (to help you diagnose after a symptom paged you), not on the pager.
On-call, runbooks, and incident response
Someone is always on-call: a designated engineer (the role rotates through the team on a schedule so the burden is shared) who is reachable and responsible for responding when a page fires. To make 2am decisions survivable, each alert links to a runbook — a short, practical document that says "if this alert fires, here is what it usually means, the first things to check, and the exact commands to run." A good runbook lets a half-asleep engineer who didn't write the service still take the right first steps, instead of waking three other people. When an incident actually hits, the order of operations is fixed and important:
detect ─▶ an alert (or a user) tells you something's wrong
mitigate ─▶ STOP the bleeding first — roll back, fail over, shed load
resolve ─▶ THEN find and fix the real root cause, calmlyThe single most counterintuitive rule here: mitigate before you diagnose. The instinct of most engineers is to figure out why it broke first — but every minute spent investigating is a minute users keep suffering. Stop the bleeding first; understand it later. The fastest mitigation is usually a rollback — redeploying the last known-good version — because the most common cause of a new incident is the most recent change. (Other mitigations: fail over to a healthy replica or region, or shed load by dropping low-priority traffic so the core keeps working.)
This is exactly why fast, automated deploys and one-click rollbacks (CI/CD) are a reliability tool, not just a developer convenience — if a rollback takes an hour, your mitigation is an hour, and that's an hour of outage you didn't need. And in a world of many cooperating services, the failure is rarely where the page fired: a slow database makes a service time out, which makes its callers pile up retries, which takes down a service that looked totally innocent. See Distributed systems in practice for these cascading failures, retry storms, and circuit breakers — the patterns that explain why "the page said auth, but auth was fine."
Blameless postmortems
After a real incident, the team writes a postmortem: a document analyzing what happened so the same failure can't recur. It is blameless — it deliberately never names a person as the cause. The premise is structural: if one engineer running one wrong command could take down production, then the system let it happen — there was no guardrail, no review, no confirmation prompt, a confusing tool, a missing safety check. So you fix the system, not the person. A blameless postmortem contains, at minimum: a timeline (what happened and when, from first symptom to full recovery, including detection and mitigation times), the root cause (the underlying systemic reason, often found by repeatedly asking "but why did that happen?"), the impact (how many users, how long, budget consumed), and concrete action items with owners (the specific fixes that prevent recurrence). Why blameless? Because a blameful culture punishes the person who hit the bug, so people learn to hide mistakes and near-misses — which means fewer reports, fewer fixed weaknesses, and ultimately more outages. Psychological safety is a reliability feature: you get more honest reports, so you fix more real weaknesses.