Designing evals for LLM systems
As of June 2026: eval tooling and benchmark names cited below reflect that date — confirm before relying on them.
📖 Walk me through it — plain English
An "eval" (short for evaluation) is just a test for an AI feature — a repeatable measurement of how good its output is. When you write normal software, a function that adds two numbers always returns the same answer, so you write a unit test: input 2 and 3, expect 5. But an "LLM" (Large Language Model — the AI behind tools like ChatGPT or Claude) is fuzzy: ask it to summarize an article twice and you get two different wordings, both maybe fine. The technical word for this is non-deterministic — the same input can produce different outputs. So you can't just check "did it return exactly this string." This lesson is about how you build trustworthy tests for that fuzzy behavior, and in an interview you'll be asked to design exactly that on the spot ("build me an eval for this summarizer").
Two one-line ideas anchor everything below. First: you can't improve what you can't measure. If you tweak a prompt and have no eval, you're guessing whether you made things better or worse. Second: because LLM output is non-deterministic, a single eyeball check ("looks good to me") proves nothing — it might have been luck. You need a structured set of measurements you can re-run. People call working this way eval-driven development: you build the eval first, then change the model or prompt, then let the eval tell you if it helped — the AI version of test-driven development.
Everyday analogy: imagine grading a class of essays instead of a math quiz. A math quiz has one right answer per question, so you grade it with a stencil (fast, objective). Essays have no single right answer, so you need a rubric and a careful human reader. Evals are the same split — some AI outputs you can grade with a "stencil," and some you need a "judge" with a rubric. A good eval system uses both, picking the cheap stencil whenever it can.
The big idea is to match each kind of check to each kind of question. The lesson's menu, in plain terms:
- Golden set — your answer key, also called a golden dataset or ground truth ("ground truth" just means the known-correct answer you trust). A hand-picked pile of (input, the answer we wanted) pairs, usually 50–500 of them. This is the "truth" you measure everything against.
- Programmatic checks — the stencil. Plain code asks yes/no questions: does the output exactly match? Is it valid JSON? Did it call the right tool? Cheap and gives the same verdict every time ("deterministic").
- LLM-as-judge — hand the essay to a strong AI and have it score the output against a rubric (a written scoring guide: "give 1 point for accuracy, 1 for brevity…"). Used when quality is subjective (is this summary good? is this reply helpful?).
- Pairwise comparison — instead of "rate this 1–10," ask "is A or B better?" People and AI judges are far more consistent comparing two things than scoring one in a vacuum.
- Online sampling — once it's live, grade a small slice (say 1%) of real traffic so you notice if quality slowly slips ("drift").
- User feedback signals — free real-world grades: thumbs up/down, or how much the user edited what the AI produced before keeping it.
How to actually answer this in an interview, step by step:
- Build a golden set from real traffic, and deliberately over-stuff it with the hard 10% — weird inputs, edge cases, the stuff that breaks. Easy cases everyone passes; they teach you nothing.
- Use the cheap check first. Anything code can verify (format, exact answer, tool called) goes to a programmatic check. Only the subjective parts go to an LLM-as-judge.
- Watch the judge's biases. An AI judge tends to prefer whatever it sees first ("position bias"), longer answers ("length bias"), and answers written by its own model family ("self-enhancement"). Fixes: shuffle the order, normalize length, and use a different model family as the judge.
- Report a range, not a point. A 100-example test is noisy. Say "82 ± 4%" (a confidence interval) instead of a flat "82%," which pretends to a precision you don't have.
- Tier by cost. Tiny fast checks on every prompt tweak, the full suite when you swap models, the slow expensive evals weekly.
Why this is the right instinct: it reframes AI from "magic we hope works" into "a system we measure." Cheap objective checks catch the easy regressions for almost nothing; judges and human-labeled subsets cover the squishy parts; confidence intervals keep you honest; and because real-world inputs drift, you refresh the golden set so your answer key never goes stale. That's the whole template the lesson hands you: a ~100-item golden set plus ~20 adversarial cases, programmatic checks where possible and a different-family LLM judge where not, tracked with a ± range on every change.
First-class interview round at Anthropic, OpenAI, Sierra, and most AI startups. You're asked to design an eval harness for some LLM behavior — "evaluate this summarizer" or "build the eval for our agent." Tests whether you treat LLM systems as engineering, not magic.
Of all the AI-engineering-loop skills, evals are the single most weighted and most failed. The reason is structural: evals are how a team knows anything they ship actually works, so they're the closest proxy for "can this person ship AI responsibly" — and most candidates have done RAG and agents but never built a real measurement harness, so they fold here. The concrete bar: if you cannot whiteboard an LLM-as-judge eval on the spot — a golden set, a rubric, a different-family judge, a metric with a confidence interval — it reads as a near-automatic reject at an AI-eng loop, no matter how strong your coding was. Treat this lesson as the highest-leverage one in the phase.
First, the vocabulary (define every term once)
Eval talk is dense with jargon. Here is the whole vocabulary in plain English so nothing below is a mystery:
- Eval / evaluation — a repeatable test that measures how good an AI's output is, expressed as a number (or set of numbers) you can track over time.
- Ground truth — the answer you trust as correct. For "what's the capital of France?" the ground truth is "Paris." It's the yardstick you compare model output against.
- Golden dataset / golden set — a curated collection of (input → ground-truth answer) pairs. Your answer key.
- Offline eval — testing against your saved golden set, in the lab, before shipping. Repeatable and safe. Online eval — measuring quality on live production traffic after shipping. Realistic but you can't re-run a given day. You need both: offline catches regressions before users see them; online catches drift offline missed.
- Precision — of the things the model flagged as positive, what fraction were actually right? (Few false alarms.) Recall — of all the things that should have been flagged, what fraction did the model catch? (Few misses.) F1 — a single number that balances the two (their harmonic mean), so you can't win by acing one and tanking the other.
- LLM-as-judge — using a strong language model to grade another model's output against a rubric, instead of (or before) a human.
- Rubric — the written scoring guide you hand the judge (human or LLM): the explicit criteria and how many points each is worth.
- Regression suite — the fixed battery of evals you re-run on every change to make sure you didn't break ("regress") something that used to work.
- A/B test — ship version A to half of real users and version B to the other half, then compare a real metric (thumbs-up rate, conversions). The gold standard for "did this actually help users?"
- Human eval — people read outputs and score them by hand. Most trustworthy, slowest and most expensive.
- Hallucination rate — the fraction of outputs that contain confident, made-up, false statements. A key safety/quality metric for any system that states facts.
- Eval-driven development — writing the eval first, then changing the system, and letting the eval decide whether the change is an improvement.
- Golden set — hand-curated (input, expected_output) pairs. 50–500 examples. The ground truth.
- Programmatic checks — exact match, regex, JSON-schema validity, "did it call tool X." Cheap, deterministic.
- LLM-as-judge — a stronger model scores outputs on a rubric. For subjective tasks (summary quality, helpfulness).
- Pairwise comparison — "is A or B better?" More reliable than absolute scoring.
- Online sampling — score X% of production traffic; track drift over time.
- User feedback signals — thumbs, edit-distance from output to what user kept, conversion.
A worked example: building an eval for a support-reply bot
Abstract advice is forgettable, so let's build one concrete eval end to end. Say the task is a bot that, given a customer support email, drafts a reply. Two parts of "good" matter: it must be correct (no made-up policies, right facts) and it must be well-written (polite, on-topic, the right length). That split tells us which tool to use for each part.
Step 1 — assemble the golden set. Pull ~100 real support emails from logs, and for each one have a senior support agent write (or approve) the ideal reply. That approved reply is the ground truth. Deliberately include ~20 nasty cases: angry customers, requests the bot should refuse, questions with no answer in the docs. Those adversarial cases are where quality actually breaks.
Step 2 — write the cheap programmatic checks. Anything code can verify deterministically goes here, run on every single output:
def programmatic_checks(reply, case):
checks = {}
checks["not_empty"] = len(reply.strip()) > 0
checks["length_ok"] = 20 <= len(reply.split()) <= 200
checks["no_placeholder"] = "[INSERT" not in reply # model left a template hole
# refusal cases MUST refuse — a cheap, objective correctness check
if case["should_refuse"]:
checks["refused"] = any(w in reply.lower() for w in ["can\'t", "unable", "cannot"])
return checks # all True = passed the stencil; same verdict every runStep 3 — define the metric for the squishy part. "Is this reply helpful and on-topic?" can't be regexed, so send it to an LLM-as-judge with an explicit rubric, and ask for a number plus a short reason (the reason lets you audit the judge later):
JUDGE_PROMPT = """You are grading a support reply against the approved answer.
Score 1-5 on each, then return JSON {accuracy, tone, total, reason}:
accuracy: does it match the approved facts, with nothing invented?
tone: polite, professional, addresses the customer's actual question?
A reply that invents a policy not in the approved answer scores accuracy=1."""
# Use a DIFFERENT model family as judge than the one that wrote the reply,
# so the judge can't quietly reward its own house style (self-enhancement).
score = judge_model(JUDGE_PROMPT, approved=case["truth"], candidate=reply)Step 4 — roll it up into one trackable number with a range. Run all 100 cases, compute the pass rate for programmatic checks and the average judge score, and report each with a confidence interval (the ± range), because 100 samples is noisy:
prog_pass = mean(all_checks_passed) # e.g. 0.94
judge_avg = mean(total_scores) / 5 # normalize to 0-1, e.g. 0.81
ci = 1.96 * std(scores) / len(scores)**0.5 # 95% confidence interval
print(f"prog {prog_pass:.0%} | judge {judge_avg:.0%} ± {ci:.0%}")
# prog 94% | judge 81% ± 4% <- now a prompt tweak that moves judge to 82% is NOISEThat's a complete, self-contained eval: a golden set, a cheap objective layer, a rubric-driven judge for the subjective layer, and a headline metric with honest error bars. Everything else in this lesson is about doing each of those four steps without fooling yourself.
Position bias (prefers first option), length bias (prefers longer), self-enhancement (prefers same-model outputs). Mitigations: randomize position, normalize length, use a different model family as judge.
Production traffic drifts. Refresh the golden set quarterly with new failure modes from real traffic.
Fast smoke tests on every prompt change. Full eval suite on every model swap. Slow expensive evals weekly.
Your golden set should over-index on the hard 10% — adversarial inputs, edge cases, known-confusing patterns. Easy cases tell you nothing.
100-sample eval has noisy metrics. Run with N samples and report ±CI. "Accuracy 82 ± 4%" is a real number; "Accuracy 82%" lies.
Have 2–3 humans label a subset; measure inter-annotator agreement. If humans disagree 30%, your model can't do better than 70% on objective metrics.
BLEU/ROUGE for summaries miss semantic quality. Combine metrics + spot-check.
Automated vs human evals: the core tradeoff
Every eval choice trades cost and speed against trustworthiness. The three rungs, cheapest to most trusted:
Fractions of a cent, instant, perfectly repeatable. But only grades what code can verify — format, exact match, tool calls. Blind to whether prose is good.
Cheap-ish (a model call per item), fast, scales to thousands. Handles subjective quality. But it has biases and can be flat wrong — calibrate it against humans before you trust it.
Slow (minutes per item), expensive, doesn't scale. The most trusted signal and the only true ground truth for taste. Reserve it for a small subset and for calibrating the judge.
Programmatic for everything it can grade, LLM-as-judge for the rest, and a small human-labeled subset to check the judge agrees with humans. If the judge tracks humans, trust it at scale; if not, fix the rubric.
Pitfalls that quietly ruin an eval
- Tiny or biased eval sets. Twenty examples scraped from one happy customer tell you nothing about the angry ones. A small set has wide confidence intervals (every result is "within noise") and a skewed set measures the wrong population. Aim for ~100+ items that mirror real traffic plus deliberate hard cases.
- Gaming the metric (Goodhart's law). "When a measure becomes a target, it stops being a good measure." Optimize hard for ROUGE and the model learns to copy phrases without understanding; optimize for judge score and it learns the judge loves long, flattering answers. Defense: use several metrics, keep a human spot-check, and refresh the set so the model can't overfit a frozen test.
- Trusting an uncalibrated judge. An LLM judge that nobody compared to humans is just a confident guess. Always validate it against a human-labeled subset first.
- Mistaking noise for signal. +2 points on n=100 is usually within the ± range. Without a confidence interval you'll ship changes that did nothing — the single most common AI-product engineering error.
- A stale golden set. Real inputs drift; an answer key from last quarter slowly stops reflecting today's traffic. Refresh it with fresh failure modes on a schedule.
Takeaway: you can't improve what you can't measure, and LLM output is non-deterministic, so trustworthy evals are the whole game. Match the tool to the question — cheap deterministic checks for anything code can verify, an LLM-as-judge with a clear rubric for subjective quality, and a small human-labeled subset to calibrate that judge. Over-index your golden set on hard cases, report every metric with a ± confidence interval, refresh the set as traffic drifts, and never optimize a single number you could game. That is eval-driven development: build the measurement first, then let it tell you whether your change actually helped.
Go deeper (optional): for hands-on patterns, see OpenAI's open-source evals framework and the LMSYS Chatbot Arena papers on pairwise human preference and "LLM-as-a-judge" bias. The agent-eval relationship is covered in the agent design lesson.