System design — AI customer-support copilot

As of June 2026: model choices, vendor stacks, and cost figures cited below reflect that date — confirm before relying on them.

The Sierra / Decagon / Intercom Fin / Zendesk AI canonical question. "Design a copilot that drafts replies for support agents (or auto-resolves tickets) from our help center + ticket history." Tests whether you compose RAG + agent + evals + cost + safety into one production system — the AI-flavored sibling of the URL-shortener round.

Before anything: the on-ramp

If you have never built one of these, here is the whole thing in one breath. A customer types a message ("where is my order?"). Your system finds the few help-center passages and account facts that bear on that message, hands them plus the message to an AI, and the AI writes a reply that quotes those passages. A set of automatic checks reads the reply before anyone sees it. If the system is confident and the checks pass, the reply goes out (or goes to a human agent to approve, depending on your risk appetite). Every reply is recorded so you can grade it later and keep score. That is it. The rest of this lesson is just naming each step carefully and explaining the choices inside it.

Why not just ask the AI directly, with no retrieval? Because an LLM only knows what was in its training data — it has never seen your refund policy, your current shipping times, or this customer's order. Ask it cold and it will hallucinate: produce a confident, fluent answer that is simply invented. Grounding — forcing the model to answer only from documents you supplied, and to cite them — is the cure, and that is exactly what RAG gives you. Hold onto that one idea; the entire architecture exists to make answers grounded, safe, cheap, and measurable.

The vocabulary, defined once

These words recur for the rest of the lesson. Read them slowly now and the architecture will read like plain English afterward.

• RAG (retrieval-augmented generation) — the pattern of retrieve, then generate. Instead of trusting the model's memory, you look up relevant text first and feed it in. "Augmented" = the generation step is helped by retrieved facts.
• Embedding — a function that turns text into a fixed list of numbers (a vector, e.g. 1,536 floats) capturing its meaning. Texts about the same idea land near each other in this number-space, so "nearness of vectors" becomes "similarity of meaning."
• Vector database — a store built to answer "given this query vector, which stored vectors are closest?" in milliseconds over millions of entries. It is your AI's searchable memory of the knowledge base.
• Chunking — cutting long documents into bite-size passages (say 200–800 tokens) before embedding. You retrieve and cite at the chunk level, so chunk size is a real tuning knob: too big wastes prompt space and blurs relevance; too small loses context.
• Retrieval — the lookup step that, given the user's message, returns the most relevant chunks. Quality of retrieval caps quality of the whole system: the model cannot cite what you never fetched ("garbage in, garbage out").
• Reranking — after a fast first pass returns ~50 candidate chunks, a slower, smarter model re-scores them for true relevance to this query and keeps the best ~5. It trades a little latency for a large jump in precision.
• Grounding / citations — requiring the answer to come from the supplied chunks, and to point back at them ("per article #142…"). Citations let humans verify and make hallucinations easy to catch.
• Hallucination — a fluent but false statement the model invents when it lacks the fact. The #1 failure mode of any LLM product.
• Guardrails — deterministic (non-AI) checks wrapped around the model: PII redaction, jailbreak detection, forbidden-phrase blocking, hard caps on risky actions. They catch what prompts alone cannot guarantee.
• Human handoff / escalation — the path that routes a ticket to a real agent when the system is unsure, the action is risky, or the customer asks. Always design the off-ramp, not just the happy path.
• Latency budget — your time allowance from message to reply (chat: ~1–3s feels live; email: seconds-to-minutes is fine). You spend it across retrieval + rerank + generation, so the budget drives model and step choices.
• Caching — reusing work across calls. Prompt caching reuses the unchanged prefix (system prompt + policy + docs) so you stop paying to re-process it; a response cache reuses a full answer for repeated identical questions.
• Eval / feedback loop — the measurement system: log every interaction, grade a sample (by an LLM-judge and by humans), watch metrics, and feed what you learn back into prompts, retrieval, and the confidence threshold. No eval = flying blind.

Why open with questions instead of a diagram? Because the answers change the architecture, not just the parameters. Auto-resolve means a wrong reply reaches a real customer with no human safety net, so it demands stronger guardrails, a higher confidence threshold, and tighter caps on tool actions; suggest-to-human lets a person catch mistakes, so you can be more aggressive. Channel sets your latency budget — a live-chat user staring at a typing indicator needs a reply in a second or two, while an email can take a minute, which lets you afford reranking and a bigger model. Scale decides whether pgvector in your existing Postgres is enough or you need a dedicated vector store, and whether cost math matters. Tools decide whether this is a read-only answer-bot or an agent that can move money. The success metric tells you which number you are actually optimizing. Stating these tradeoffs is itself senior signal — you are showing you know the design forks before you commit.

Walking the architecture, station by station

The numbered list above is the skeleton. Below is the same flow with the reasoning filled in, mapped back to the data → retrieval → generation → eval → guardrails → ops rubric so you can see which peg each step hangs on.

Step 1 — Data: ingestion (the offline pipeline)

Everything starts with knowledge the model can draw on. Your sources are the help center articles (the official policies) and past resolved tickets (how real problems were actually solved, in your real voice). This runs offline — ahead of time, on a schedule — not while a customer waits. The pipeline is: load each document, chunk it into passages, run each passage through an embedding model to get its vector, and write the vector plus its source text and metadata (article ID, last-updated date, language, tenant) into the vector DB. For a B2B product you isolate each customer in their own namespace so company A's tickets can never surface company B's documents — a hard tenancy boundary, not a "be careful in the prompt" boundary.

The non-obvious requirement is freshness. A knowledge base is not write-once: policies change, prices change, products launch. If you embed the docs once and never re-ingest, your bot will confidently quote last quarter's return policy — a stale knowledge base, which is hallucination's quieter cousin (the source is real, it's just out of date). So ingestion must be a recurring job triggered whenever a doc changes, and chunks should carry a last-updated timestamp you can filter and rank on.

# OFFLINE ingestion — runs on doc change, not per ticket
for doc in help_center_docs + resolved_tickets:
    for chunk in split(doc.text, size=500, overlap=50):   # passages w/ slight overlap so context isn't cut mid-thought
        vec = embed(chunk.text)                            # text -> list of numbers (meaning)
        vector_db.upsert(
            id=chunk.id, vector=vec, text=chunk.text,
            tenant=doc.tenant, updated_at=doc.updated_at,    # metadata for filtering + freshness
        )

Step 2 — Retrieval: finding the right slice at query time

Now a ticket arrives. First a cheap intent classifier (a small, fast model) labels it — FAQ, account-action, billing, escalation — so you can route: a plain FAQ doesn't need your most expensive model, while "I want to cancel and I'm furious" routes straight to a human. Then comes retrieval, and the load-bearing decision is hybrid search. You run two searches in parallel: dense (embed the ticket, find nearest chunk vectors — great at meaning, e.g. "my package never showed up" matches a "delayed delivery" article even with no shared words) and sparse / BM25 (classic keyword scoring — great at exact strings like an order ID ORD-44291, an error code, or a SKU that embeddings smear into a vague blob). You fuse the two ranked lists (reciprocal rank fusion is the standard trick), then rerank the top ~50 down to the top ~5 with a model that judges true relevance to this query. Filter by tenant (security), recency (freshness), and channel before you hand the survivors to the generator.

Say "hybrid + rerank" unprompted — it is the production default and the most common thing junior answers miss. Pure dense retrieval alone is the single biggest practical bug in support bots because customers' messages are full of exact tokens that vector search blurs.

# ONLINE retrieval — runs per ticket, inside the latency budget
intent = classify(ticket)                              # cheap model: FAQ / billing / escalation ...
dense  = vector_db.search(embed(ticket.text), k=50)   # meaning match
sparse = bm25.search(ticket.text, k=50)              # exact-token match (order IDs, codes)
fused  = reciprocal_rank_fusion(dense, sparse)         # combine both rankings
top5   = rerank(ticket.text, fused)[:5]              # smarter model re-sorts; keep the best few

Step 3 — Tools: when the answer needs live facts or actions

Retrieval pulls knowledge; tools pull state and perform actions. "Where is my order?" can't be answered from a help article — the model needs to call get_order(id) against the live order DB. This is what turns a passive answer-bot into an agent: you expose a small, explicit menu of functions — get_order, issue_refund, escalate_to_human, request_clarification — and the model decides which to call. The rule that matters: reads are cheap, writes are dangerous. Reading an order is harmless; issuing a $10,000 refund is not. So every state-changing tool is wrapped in deterministic policy — small refunds may auto-execute, anything over a threshold returns "pending human approval." The model proposes; your code disposes. Never let the prompt alone be the thing standing between a clever message and your bank account.

Step 4 — Generation: assembling the prompt and drafting

Now you build the prompt the model will actually see. Think of it as a layered sandwich, top to bottom: system / persona (who the bot is, the tone, the hard rules — "never promise a refund you can't authorize"), the per-tenant policy (this customer's specific rules), the retrieved chunks from Step 2 (the grounding material, each tagged with its source so the model can cite it), any tool results from Step 3 (the live order facts), and finally the customer's message at the bottom. You instruct the model to answer only from the supplied material and to attach citations pointing back to the chunks. That ordering is deliberate and it is also a caching decision: the top of the sandwich (system + policy) barely changes across a conversation, so it's the perfect cache prefix — more on that under ops.

Which model? Match it to intent, not ego. A frontier (top-tier) model gives you nuance and tone for refund judgments and tricky complaints; a small fast model is plenty for an FAQ. Routing every ticket to your biggest model is the lazy, expensive choice — cascade instead (cheap model first, escalate to the big one only when needed).

# Prompt assembly — stable prefix first (cache it), volatile message last
prompt = [
  system_persona,            # tone + hard rules     (rarely changes -> cacheable)
  tenant_policy,             # this customer's rules (per-conversation -> cacheable)
  format_chunks(top5),       # grounding docs w/ source tags for citations
  format(tool_results),      # live order facts from get_order(...)
  ticket.text,               # the customer's actual message (changes every turn)
]
draft = generate(prompt, model=pick_model(intent), cite=True)

Step 5 — Guardrails & citations: never trust the raw output

The model's draft is a proposal, not a sent message. Between draft and customer sits the safety gate — deterministic checks that don't depend on the model behaving. On the way in: redact PII (don't ship card numbers to the LLM provider) and run a jailbreak detector for injection attempts ("ignore your instructions and…"). On the way out: a policy classifier scans the draft, a hard list blocks forbidden phrasing ("guaranteed refund," legal commitments), and the action caps from Step 3 are enforced regardless of what the model "decided." Layer these — input check, output check, action cap — so a single bypassed layer still can't cause real harm. This is defense in depth: prompt-only safety fails predictably, so you wrap the probabilistic model in deterministic walls.

Citations earn their own line. By requiring every claim to point at a retrieved chunk, you make grounding checkable: a reviewer (human or an automated faithfulness check) can confirm the answer actually follows from the source. An answer with no citation, or one whose citation doesn't support it, is your loudest hallucination signal — and a reason to drop confidence and route to a human.

Step 6 — Suggest vs send, and human escalation

Now the central decision: does this reply go out automatically, or to a human first? The output carries a confidence signal (from the classifier, model log-probabilities, whether retrieval found strong matches, whether citations check out). Above your threshold → auto-send; below → it becomes a draft the agent edits and approves. And there must always be an escalation / handoff path: the model can call escalate_to_human when it's unsure, when policy says certain topics (legal, cancellations, abuse) are human-only, or when the customer simply asks for a person. Designing the off-ramp is not optional — a copilot that traps an angry customer in a loop is worse than no copilot. The threshold itself is the product's main control knob, which the eval loop tunes; that's the next station.

Step 7 — Eval & the feedback loop: knowing it's good, and keeping it good

An AI feature you can't measure is one you can't improve or even trust. So log everything: the ticket, the retrieved chunks, the tool calls, the prompt, the draft, the confidence, whether it was sent or escalated, and the eventual outcome (did the customer come back? did CSAT drop?). Then grade a sample two ways. An LLM-judge — a second model prompted to score the first on faithfulness (did the answer stay true to the cited docs?), helpfulness, and tone — gives you cheap, continuous coverage. A weekly human review of a sample anchors the judge to ground truth and catches what automation misses. The two key product metrics are deflection (share of tickets resolved without a human) and CSAT (customer satisfaction), and they pull against each other: crank the confidence threshold down and deflection rises but CSAT falls as wrong answers slip through; crank it up and CSAT is safe but you deflect almost nothing and save no money. The feedback loop closes the circle — eval findings feed back into the prompt, the retrieval config, the chunking, and especially that threshold. That loop, not the model choice, is what makes the system get better over time.

Step 8 — Ops: cost, latency, caching, fallbacks

Last station, and the one juniors skip. Cost: argue from a number, not a vibe. A Sonnet-class model at roughly $3/M input + $15/M output, with a ~2K-token cached prompt and ~500-token reply, costs about $0.008 per ticket — about $800/day at 100K tickets. Every model and routing choice should be defended against that figure. Latency budget: in chat you have a second or two end-to-end, split across retrieval + rerank + generation, so you might skip reranking or shrink the model for the fast path; in email you have room to spare. Caching is your biggest lever on both: prompt caching reuses the unchanged sandwich-top (system + tenant policy + retrieved docs) across a multi-turn conversation so you stop re-paying to process it — a 5–10× cut on long threads — and a small response cache can short-circuit the literal-identical FAQ asked a thousand times a day. Fallbacks: assume the LLM provider will have a bad day, and design for it — degrade to retrieval-only ("here are the top 3 relevant articles; an agent will follow up") rather than going dark. Monitoring on latency, cost, deflection, CSAT, and error rates closes out ops.

The three pitfalls that sink these systems

Name these unprompted; each maps to a station you must have covered.

• Hallucination — the model invents a confident, false answer. Cured by grounding: retrieve real chunks, answer only from them, require citations, and drop confidence (route to a human) when citations are missing or don't support the claim.
• Stale knowledge base — the answer is grounded but in an out-of-date doc. Cured by treating ingestion as a recurring job, stamping chunks with updated_at, and ranking/filtering on freshness so old policies don't win.
• No eval — shipping with no way to measure quality, so regressions (from a model upgrade, a prompt tweak, tone drift) go unseen until customers complain. Cured by the eval/feedback loop: log everything, LLM-judge + weekly human review, and watch deflection-vs-CSAT.

A fourth, quieter pitfall: prompt-only safety. "We told the model not to do bad things" is not a guardrail — it's a wish. Wrap the model in deterministic checks (PII redaction, output classifier, hard action caps) so a successful injection still can't send a forbidden phrase or move real money.