📒 problems 📖 glossary
Mock interview
46:30
ready
of 46:30
Lesson 90 · Engineering craft

CI/CD — let the robots gate every change

On a real team, code is not "done" when it runs on your laptop — it is done when an automated pipeline has built it, checked it, and shipped it without a human babysitting each step. The moment more than one person touches the same codebase, two famous pains appear. The first is "it works on my machine": your code runs fine for you because your laptop happens to have the right library versions, the right environment variables, and that one file you forgot to commit — but it falls over the instant a teammate or a server tries to run it. The second is "merge hell": five people each work in isolation for two weeks, and when they finally try to combine their changes, the edits collide everywhere and untangling them takes days. CI/CD is the discipline — and the automation — that kills both pains. It is a chain of automatic gates that run on every push, so breakage is caught before it reaches your teammates or your users. Master this and you stop being the person who breaks main.

The vocabulary first

Five words do all the work in this lesson, so let us pin them down before anything else:

  • A build is the act of turning your source code into something runnable — compiling it, bundling it, installing its dependencies. "The build failed" means that transformation did not even complete, never mind whether the tests pass.
  • An artifact is the concrete, finished output of a build that you can store and ship: a compiled binary, a zipped folder of web files, a container image. You build the artifact once and deploy that same artifact to every environment, so what you tested is exactly what runs in production.
  • A pipeline is the whole automated sequence — the ordered list of steps that runs from "someone pushed code" all the way to "it is live."
  • A stage (or step/job) is one link in that chain: build, lint, test, deploy. Each stage has a job to do and either passes or fails.
  • A runner is the machine — usually a fresh, throwaway virtual machine or container spun up by your CI provider — that actually executes a stage. "Fresh" is the point: it starts clean every time, with nothing left over from your laptop, which is precisely why it exposes "works on my machine" problems.

CI vs CD vs Continuous Deployment, untangled

These three terms get used loosely and mean genuinely different things. The difference is entirely about how far the automation goes and whether a human still pushes a button.

  • Continuous Integration (CI) — every push is automatically built and tested against the shared branch, many times a day, so small changes integrate constantly instead of piling up. What is automated: build + checks + tests. What a human does: reviews and merges the pull request. Goal: main is always green and merges stay small. Example: you open a PR; within four minutes a bot comments "build passed, 412 tests green," a teammate approves, you merge. Nobody waited two weeks; nothing rotted.
  • Continuous Delivery (CD) — every green build is automatically packaged into an artifact and pushed to a staging environment, kept perpetually ready to release. What is automated: everything up to and including deploy-to-staging. What a human does: clicks one "Release to production" button when the business is ready. Example: Friday's green build sits on staging; the product manager clicks "Release" on Tuesday after a demo, and it goes live with zero extra work.
  • Continuous Deployment — same as Delivery, but there is no button: a green build flows straight to production by itself. What is automated: the entire path, including production. What a human does: nothing, for a normal change. Example: you merge a one-line fix at 2pm; by 2:09pm it is serving real users, because the pipeline trusted its own gates. This requires deep confidence in your tests and your monitoring, because there is no last human to catch a mistake.

The confusing part is that "CD" can stand for either Delivery or Deployment. The reliable way to keep them apart: Delivery stops at a manual approval gate; Deployment removes that gate. Both keep main always releasable — they only differ on who, if anyone, pulls the trigger.

The pipeline is a chain of gates

The single most important mental model is this: a pipeline is a chain of gates, and each gate must pass before the next one even starts. The instant a stage goes red, the line stops — later stages never run, and the failure report points straight at the gate that broke. This is called failing fast: you want the cheapest, fastest checks first so that a typo is caught in ten seconds by the linter rather than after a twelve-minute test suite and a deploy. The first red stage stops the line; everything downstream is irrelevant until you fix it.

pushcheckoutbuildlinttypechecktestssecurity scanbuild artifactdeploy staging(approve?)production

Walk each gate and what it catches:

  • Checkout — the runner pulls a clean copy of your exact commit from version control. (See Version control on a team for how that commit got there.) Catches: nothing yet — it just sets up a pristine starting point with none of your laptop's leftovers.
  • Build — install dependencies and compile/bundle the code. Catches: missing files you forgot to commit, version mismatches, syntax errors that break compilation. This is where most "works on my machine" problems surface, because the runner is clean.
  • Lint — a linter is a tool that scans source code for stylistic issues and bug-prone patterns without running it: unused variables, missing await, shadowed names, inconsistent formatting. Catches: the small, mechanical mistakes that humans waste review time on.
  • Typechecktype checking verifies that values are used consistently with their declared types (you do not pass a string where a number is expected, or call a method that does not exist). Catches: whole classes of errors before the code ever runs — the kind that would otherwise blow up at runtime in front of a user.
  • Tests — runs your automated test suite; any single red test fails the whole gate. This is the heart of the pipeline. See Testing fluency for what actually goes in here (unit, integration, end-to-end). Catches: behaviour that changed when it should not have — the actual logic regressions.
  • Security / dependency scan — automatically checks your dependencies against databases of known vulnerabilities, and may scan your own code for risky patterns and leaked secrets. Catches: a library version with a published exploit, an API key accidentally committed.
  • Build artifact — package the verified code into the immutable thing you will deploy (a container image, a zip). Catches: packaging problems; produces the one artifact promoted unchanged through every later environment.
  • Deploy to staging — push that artifact to a production-like environment that real users cannot see. Catches: deployment and configuration problems, and lets humans click around in something close to the real thing.
  • Manual approval — the optional human gate. Present in Continuous Delivery, absent in Continuous Deployment. Catches: business-timing problems ("not during the holiday freeze").
  • Production — the same artifact goes live to real users. After this, your eyes move to observability — you watch the deploy, not assume it worked.

Pipelines are defined as code

You do not configure a pipeline by clicking buttons in a web UI — you write it in a YAML file that lives in the repo, so the build is versioned, reviewed, and rolls back alongside the app. The exact keys differ by vendor (GitHub Actions, GitLab CI, CircleCI), but the shape is universal. Read this one stage by stage; the annotations explain every line:

pipeline.yml (generic)
on: push                   # trigger: run this whole pipeline on every push

stages:
  - name: build
    run: npm ci && npm run build      # clean-install deps, then compile/bundle
  - name: check
    run: npm run lint && npm run typecheck   # style/bug patterns, then type errors
  - name: test
    run: npm test            # fails the pipeline on any red test
  - name: scan
    run: audit-deps --fail-on=high   # block known-vulnerable dependencies
  - name: deploy-staging
    run: deploy ./dist --env=staging # auto-ship the artifact to staging
  - name: deploy-prod
    needs_approval: true   # human clicks "release" → this is the CD gate
    run: deploy ./dist --env=production

Because stages run in order and stop at the first failure, a broken lint never reaches test, and a red test never reaches deploy-staging. Remove the needs_approval: true line and you have just converted Continuous Delivery into Continuous Deployment.

Why fast pipelines matter

Pipeline speed is not a nicety — it changes behaviour. The whole value of CI comes from small batches and fast feedback: tiny changes integrated constantly, each verified in minutes, so a failure has a small "blast radius" (only a few lines could be the culprit). Make the loop slow and people quietly stop using it.

Concretely: with a 4-minute pipeline, you push, glance away, and it is green before you have switched tasks — so you push often, in small pieces, and bugs are caught the same minute you wrote them. With a 40-minute pipeline, you cannot afford to wait, so you batch up many changes to "make it worth a run," push less often, and start merging while it is still running. Now a failure could be hidden anywhere in a huge batch, the feedback arrives after you have mentally moved on, and people begin skipping or working around CI entirely. Same gates, opposite outcomes — the only difference is speed.

Deployment strategies in depth

Getting a green artifact built is only half the job; how you replace the running version matters just as much, because that is where users get hurt if something is wrong. Four strategies, each with how it works, when to reach for it, and — crucially — how you roll back.

Rolling deployment.

How it works: you run several copies (instances) of your app behind a load balancer; the deploy replaces them a few at a time — take two down, bring two up on the new version, repeat — until all run the new code. When to use: the default for most services; cheap, because you never need double the machines. Rollback: roll the same process backward, swapping instances back to the old version a batch at a time. The catch is that during the rollout, old and new versions serve traffic simultaneously, so the two versions must be compatible with each other and with the database.

Blue-green deployment.

How it works: you stand up a complete second copy of the environment (green) running the new version, while the current one (blue) keeps serving everyone. When green looks healthy, you flip all traffic from blue to green at once. When to use: when you want a clean, instant cutover and can afford to run two full environments briefly. Rollback: the fastest of all — blue is still sitting there untouched, so you flip traffic back in seconds.

before: users → [ BLUE v1 ] ( GREEN v2 warming up )
flip: users → [ GREEN v2 ] ( BLUE v1 kept idle as instant rollback )
Canary deployment.

How it works: you route a tiny slice of real traffic — say 1% — to the new version while everyone else stays on the old one, then watch error rates and latency. If the canary stays healthy you ramp to 5%, 25%, 100%; if it misbehaves you stop. (The name comes from the canary miners carried underground to detect bad air before it killed anyone.) When to use: risky changes where you want production-scale validation with minimal exposure. Rollback: route that 1% back to the old version — only a sliver of users ever saw the problem.

step 1: 99% → [ v1 ] 1% → [ v2 ] ← watch metrics
step 2: 75% → [ v1 ] 25% → [ v2 ] ← still healthy? ramp up
step 3: 0% → [ v1 ] 100% → [ v2 ] ← full rollout
Feature flags.

How it works: you ship the new code to production with it wrapped in a conditional that is switched off ("dark"), then turn it on later by flipping a flag in a config dashboard — no redeploy. This decouples deploying code from releasing a feature. When to use: to ship incrementally, to release to internal users first, or to enable a feature for 10% of accounts. Rollback: the cleanest possible — flip the flag off and the feature vanishes instantly, with no deploy and no rebuild.

Rollback is a first-class safety tool

Notice that every strategy above was described by its rollback story. That is deliberate. The point of all this machinery is not to prevent every mistake — that is impossible — but to make recovery fast and boring. A team that can revert a bad release in thirty seconds will ship confidently and often; a team that cannot will freeze in fear and ship rarely. Rollback ties directly to observability: you deploy, you watch the dashboards and error rates, and the moment something looks wrong you roll back first and investigate afterward. "Revert now, debug later" is the mark of a reliable team.

Flaky tests are pipeline poison. A flaky test is one that passes on one run and fails on the next with no code change — usually because it depends on something nondeterministic: timing (a race condition), ordering (it assumes other tests ran first), or shared state (two tests step on the same data). Flakiness is corrosive because it teaches the team to ignore red builds: the reflex becomes "just re-run it." Once "re-run it" is automatic, a genuine failure looks identical to a flaky one and gets re-run too — so a real bug sails through and CI stops protecting anything at all. Treat a flaky test as a P1 bug: quarantine it (pull it out of the blocking suite so it cannot give false reds), then fix the nondeterminism (mock the clock, isolate state, remove order dependence) or delete it. A small trusted green suite is worth far more than a thorough but flaky one.

Where this connects

CI/CD sits in the middle of the engineering-craft story. The artifact the pipeline builds is, more and more, a container image — a sealed box holding your code and its exact environment, so "works on my machine" finally becomes "works everywhere, because everywhere runs the identical box." Once that box is live, the job is not over: you lean on observability to watch the deploy in real time and on your deploy strategy's rollback to recover fast if it goes wrong. And the test gate that anchors the whole pipeline is only as good as the tests in it — which is the subject of Testing fluency.

Takeaway in one sentence: a pipeline is a chain of fast, fail-first gates that turns every push into a verified artifact and ships it with a deployment strategy whose rollback you trust — so the team can integrate constantly and release often without a human gatekeeping each step.

Go deeper (optional): read your team's actual pipeline file, then skim the docs for the runner you use (GitHub Actions, GitLab CI, CircleCI) and Martin Fowler's articles "Continuous Integration" and "Blue Green Deployment".