Cost, caching, observability — running LLMs in production
As of June 2026: token pricing, provider features, and cost examples cited below reflect that date — confirm before relying on them.
📖 Walk me through it — plain English
This lesson is about money, not algorithms. When you build a feature on top of a large language model (LLM) — the AI that powers chatbots and assistants — you pay the model provider (like Anthropic or OpenAI) per token. A token is a chunk of text, roughly 3 to 4 characters; "hello" is about one token. Every word you send in (the prompt) and every word the model sends back costs a fraction of a cent. That sounds tiny, but multiply by millions of requests a month and it becomes a real bill — the lesson's example is a feature costing $40,000 a month. The interview question is: cut that cost in half without making the answers worse.
The one rule to internalize first: you pay per token, in and out, and you design the whole feature around that meter. Two numbers run the meter — how many tokens go in (your prompt: instructions + context + the user's question) and how many come out (the model's reply). Crucially these are priced separately, and output is the expensive one. Output tokens typically cost 3–5× more than input tokens, because the model has to generate each one (a fresh forward pass per token), whereas it reads all your input in one parallel sweep. So a verbose, chatty answer can dominate your bill even when the prompt is short. Every lever in this lesson is just a way to push one of those two numbers — or their unit price — down.
The whole point is knowing which lever to pull first. There are cheap, easy wins and there are expensive, complicated wins, and the senior signal is reaching for the cheap ones before the clever ones. Let me define the key terms as we walk down the list.
Prompt caching (lever 1, always check first): a lot of what you send the model is identical on every request — your fixed instructions ("the system prompt"), reference documents you paste in, and a few worked examples ("few-shot examples"). The provider can cache that repeated front part and charge you about 90% less for it next time. It is basically free and often cuts cost 5–10×.
Model cascading (lever 2): use a cheap, weaker model for the easy 80% of questions, and only "escalate" to the expensive, smarter ("frontier") model when the cheap one is unsure. The thing that decides "is this hard?" is itself a small classifier or a confidence score from the cheap model. The rest of the levers — trimming dead words out of the prompt ("compression"), bulk overnight discounts ("batching", 50% off for work that isn't real-time), reusing answers to near-identical questions ("response caching"), and finally training your own small model — get progressively more powerful but also more work to maintain. Fine-tuning your own model is last because it adds a whole pipeline of ops to babysit.
The second half is observability — instrumenting the feature so you can see what it's doing before you launch. Concretely: log every request's prompt, response, latency, token counts, and dollar cost (a "trace"); attach quality scores to those traces (graded by a human or by another LLM acting as judge — "LLM-as-judge"); build a dashboard of daily spend broken down by feature, model, and customer; and set alerts on latency, errors, and cost spikes. Without this you are flying blind, and the classic disaster is an "agent" (an LLM that loops, taking actions on its own) with no cap on how many times it can loop — that's how you wake up to a surprise $5,000 overnight bill.
An everyday analogy: think of it like cutting a household electricity bill. First you do the free thing — turn off the lights nobody is using (prompt caching: stop paying full price for the same text every time). Then you match the tool to the job — you don't run the industrial dryer to dry one sock; you use the small one (cascading: cheap model for easy questions). You run the dishwasher overnight on the cheap rate (batching). Only as a last resort do you rip out the walls to install new insulation — a big project that pays off but takes real effort (fine-tuning your own model). And before you change anything, you read the meter so you know where the power is actually going (observability) — otherwise you can't tell if your savings hurt anything.
How to approach the interview question:
- Start with the free win: "I'd add prompt caching on the static prefix first — usually a 5–10× cut for nothing."
- Then question the model mix: "What fraction of traffic actually needs the frontier model?" Most systems are over-provisioned on intelligence, so cascading helps.
- Hold quality fixed with evals (a fixed test set of inputs with known-good answers) and cost-cut against that bar — never cut first and check quality later.
- Talk in dollars per outcome, not per token: "$0.04 per resolved support ticket" lands far better than naming a model.
- Mention safety caps: a max number of loops and a per-user spend limit, so no single request can run away.
Why this ordering wins: the early levers (caching, cascading, trimming) are reversible, low-risk, and need no new infrastructure, so they buy the biggest savings for the least effort and risk. The late levers (response caching, fine-tuning) save more but add systems you have to maintain forever. Reaching for the cheap, boring wins before the clever, fragile ones is the senior judgment the interviewer is listening for.
Token spend is real money. Interviewers at any product team shipping LLM features ask "this feature costs $40k/month at current scale — cut it in half without hurting quality." Tests whether you've actually run something with a bill attached. Senior signal: knowing which lever to pull first.
The vocabulary, defined once
Everything below uses the same handful of words. Pin them down now so the rest reads fast.
The unit you are billed in: a chunk of text ~3–4 characters (~0.75 of an English word). 1,000 words ≈ 1,300 tokens. The provider's tokenizer decides the exact split; you never count characters, you count tokens.
Two separate meters. Input = everything you send (system prompt + context + user message). Output = what the model writes back. Output is priced 3–5× higher — it is generated one token at a time. Always estimate both; ignoring output is the #1 costing mistake.
The max tokens (input + output) a model can hold in one call — e.g. 200K. It is a capacity limit, not a flat fee: you pay for the tokens you actually put in it. Stuffing the window "because you can" means paying full input price for every token on every call.
The provider stores the repeated front of your prompt (system prompt, docs, few-shots) and bills cached input at ~90% off on later calls. A config flag, not a code rewrite. Best on RAG-shaped traffic: big static prefix, tiny dynamic suffix.
Submit many requests as one offline job; the provider runs them when it has spare capacity (within ~24h) for ~50% off. For non-realtime work only — evals, bulk classification, labeling. Trades latency for price.
Send each request to the cheapest model that can handle it. A small model (cheap, fast, weaker) takes the easy majority; a large/frontier model (expensive, slow, smarter) handles the hard tail. The router is a classifier or confidence threshold.
Train a small model to imitate a large one's outputs on your task, so you get most of the quality at a fraction of the price. A specific way to make the cheap tier in routing. Adds a training/ops pipeline.
Prompting: steer a general model with instructions/examples in the prompt — zero training cost, but those example tokens are paid on every call. Fine-tuning: bake the behavior into model weights once — upfront training + hosting cost, but shorter prompts forever. Fine-tune only at high, stable volume.
Retrieval-Augmented Generation: fetch only the few relevant document chunks and put those in the prompt, instead of pasting whole manuals. A cost lever as much as a quality one — it cuts input tokens by sending the model only what it needs.
Latency = total time to a full answer. TTFT (time-to-first-token) = how long until the first word appears — what "feels" fast when streaming. Throughput = tokens/sec the system serves. These are speed/scale metrics; they trade against cost but are not the same thing as it.
Variable = per-token API spend that scales with traffic (most LLM cost). Fixed = costs you pay regardless of volume: a fine-tuning run, a hosted/reserved endpoint, an embedding index. Fine-tuning swaps variable cost for fixed — only a win above a break-even volume.
The number that actually matters: cost per unit of value — per resolved ticket, per summarized doc — not cost per token or per model. It folds in retries, cascades, and caching, and it is what the business cares about.
- Prompt caching / prefix caching — Anthropic and OpenAI cache long static prefixes (system prompt, retrieved docs, few-shot examples). 90% discount on cached input tokens. Free 5–10× cost cut on most RAG pipelines. Always check first.
- Model cascading — cheap model handles 80% of traffic; escalate to frontier model only on low-confidence or hard cases. Routing decision is itself a classifier (or a confidence threshold from the cheap model).
- Prompt compression — shorter system prompts, fewer few-shot examples, no boilerplate. Audit your prompt — most production prompts have 30–50% dead tokens.
- Batching — Anthropic Batch API, OpenAI Batch: 50% discount for non-realtime work (overnight evals, bulk classification, data labeling).
- Response caching — semantic cache (embed query, return cached answer for near-duplicates) on high-repeat workloads. Customer support FAQs benefit hugely.
- Fine-tune a small model — for high-volume narrow tasks (classification, extraction), a fine-tuned 8B beats GPT-4 at 1/50th the cost. Last resort because it adds ops complexity.
Read the ordering as effort-vs-reward: levers 1 and 4 are config changes (no quality risk, minutes of work); 2, 3, and 5 add a little logic (a router, a trimmed prompt, a cache lookup) and need an eval gate to stay safe; lever 6 trades ongoing variable cost for fixed cost plus a permanent ops pipeline. Pull from the top; stop when you have hit the target.
Worked example: where the $40k goes, and how to halve it
Costing an LLM feature is one formula: monthly cost = requests × (input_tokens × input_price + output_tokens × output_price). Let's trace it for the support-chatbot in the prompt. Assume a RAG bot on a Sonnet-class model priced at $3 per million input tokens and $15 per million output tokens (so $0.000003/in-tok and $0.000015/out-tok), serving 2,000,000 requests/month. Per request: an 8,000-token static prefix (system prompt + retrieved chunks + few-shots), a 500-token user message, and a 700-token answer.
# Per-request token bill (no optimization)
# input = prefix + user message = 8000 + 500 = 8500 tokens
# output = 700 tokens
input_cost = 8500 * 0.000003 # = $0.0255 (8500 in-tokens)
output_cost = 700 * 0.000015 # = $0.0105 (700 out-tokens — note: ~30% of cost on 8% of tokens)
per_request = input_cost + output_cost # = $0.0360
# Monthly: requests x per-request
monthly = 2_000_000 * 0.0360 # = $72,000 ... call it ~$40k after volume discounts/shorter tails -> the targetThe shape is the lesson: the 8,000-token static prefix is 94% of the input and it is identical on every one of the 2M calls. That is a giant, blinking sign that says "cache me." Now pull the levers, each quantified against the same baseline (treat the baseline as the $40k figure the interviewer gave; the math below shows the fractions, which are what carry over):
# Lever 1 — Prompt caching the 8000-token prefix (~90% off cached input)
# input drops 8500 -> effectively 8000*0.1 + 500 = 1300 "billed" in-tokens
input_cost = 1300 * 0.000003 # = $0.0039 (was $0.0255 — an 85% cut on input)
per_request = 0.0039 + 0.0105 # = $0.0144 (was $0.0360 -> 60% total cut, alone)
# Lever 3 — Prompt compression: trim 40% of dead prefix tokens (8000 -> 4800)
# stacks with caching; less to cache, faster TTFT too
# Lever 2 — Cascading: route 80% of traffic to a small model ~10x cheaper
# blended = 0.8*(per_request/10) + 0.2*per_request ~= 0.28 * per_request (a ~72% cut)
# Lever 4 — Batching the 15% that is offline (eval/backfill): 50% off that slice
# Lever 5 — Response cache: ~25% of questions are near-duplicate FAQs -> serve freeCaching alone already roughly halves the bill (~60% off), so on this workload the interview is nearly answered by lever 1. Stack compression and cascading on top and you are comfortably past 50% with quality held fixed by your eval set. Concretely, against the $40k baseline: caching → ~$16k; add cascading on the remainder → ~$8–10k. The discipline is to apply one lever, re-measure against the eval, then decide if you even need the next one.
- Ignoring output tokens. People estimate cost from the prompt and forget the reply is priced 3–5× higher per token. A model that "thinks out loud" or returns verbose JSON can put most of your bill in the output column. Cap
max_tokens, ask for terse formats, and always estimate both meters. - No caching on a static prefix. Paying full input price for the same 8,000-token system-prompt-plus-docs on every call is the most common 5–10× overspend. If a prefix repeats, it should be cached — full stop.
- Over-large model by default. Routing all traffic to the frontier model "to be safe" overpays on the easy 80%. Match model size to task difficulty; the cheap tier usually clears the eval bar on routine requests.
- Stuffing the context window. A 200K window is capacity, not a free buffer — every token you pad in is billed at full input price on every call. Retrieve the few chunks you need (RAG); don't paste the manual.
- Optimizing cost before quality is pinned. Cutting tokens or downsizing models without an eval gate ships silent regressions. Hold quality constant first, then cut against it.
- Runaway agent loops. An agent with no max-iterations and no per-user budget cap is the classic $5k-overnight surprise. Hard caps on both, every time.
- Per-request trace — prompt, response, latency, input/output tokens, model, cost. Without this you're flying blind.
- Eval scores tied to traces — every prod trace gets sampled into an eval queue (LLM-as-judge or human label).
- Tools — Braintrust (evals + traces), Langfuse (open-source traces + evals), LangSmith (LangChain-native), Helicone (proxy-based cost tracking). Pick one; don't roll your own on day 1.
- Cost dashboard — daily spend by feature, by model, by customer. Surprises kill margins.
- Alerting — p95 latency, error rate, daily spend deltas. Same shape as any prod service.
The reason observability comes paired with cost-cutting: you cannot tell whether a lever helped without per-request token/cost/quality on the same trace. The trace tells you where the tokens go (so you know which lever to pull); the eval score tells you whether the cut hurt quality; the dashboard catches the regression or spend spike before the monthly invoice does.
The harder prompt: "design an LLM serving path"
Everything above treats the model as a metered API you call. But at a frontier lab or any team that self-hosts a model, the system-design prompt flips: "design the serving path that runs this model on our own GPUs." Now you own the throughput and latency you were just buying. This is a real interview question, and it rests on a handful of durable ideas — the mechanisms outlive any specific framework name.
- Prefill vs decode — two different bottlenecks. Serving a request has two phases. Prefill reads the whole prompt in one parallel pass — it's compute-bound (you saturate the GPU's math units). Decode then emits the answer one token at a time, each token a fresh pass over the weights + cache — it's memory-bandwidth-bound (you're limited by how fast you can read memory, not by math). Knowing which phase dominates tells you what to optimize.
- KV cache — why generation is stateful. Each generated token attends to every earlier token, so the model stores per-token key/value vectors (the "KV cache") and appends to it each step instead of recomputing. The catch: that cache grows linearly with sequence length × batch size and can rival or exceed the model weights in GPU memory — so KV-cache memory, not compute, is usually what caps how many requests fit on a GPU at once.
- Continuous (iteration-level) batching — the throughput win. Naively you'd batch N requests, run them together, and wait for the slowest to finish before starting more — the GPU idles on the stragglers. Continuous batching schedules at the token level: as soon as one request finishes a step (or completes), a waiting request takes its slot. The GPU stays full, and throughput jumps several-fold over static batching. This is the single biggest single-GPU throughput lever.
- Paged KV cache — fit more on the GPU. Storing each request's KV cache in one contiguous block wastes memory to fragmentation and over-allocation. Paging it into fixed-size blocks (the virtual-memory trick, à la PagedAttention) cuts that waste to near-zero, so you fit a bigger batch — which feeds straight back into continuous-batching throughput.
- The metrics you trade. TTFT (time-to-first-token) is dominated by prefill and queue wait — what feels responsive when streaming. Throughput (tokens/sec across all requests) is what bigger batches buy. They trade: a fuller batch raises throughput but can delay any one request's first token. You size the batch to the product's latency SLO, not to max throughput blindly. Quantization (serving smaller-precision weights) and prefix caching (reuse a shared prompt prefix's KV across requests) are the other two standard levers.
The interview answer shape: "I'd put a queue in front of a continuous-batching scheduler, store the KV cache paged so I can pack a large batch, and size the batch to the latency SLO — prefill is compute-bound so it sets TTFT, decode is memory-bound so the KV cache caps my batch size. I'd add prefix caching for shared system prompts and reach for quantization if I'm memory-limited." That answer shows you understand why the levers exist, which is the whole point — the framework that implements them (vLLM and friends) is a name you can drop, not the substance.
System prompt + retrieved chunks + few-shots are static across users. That's where prompt caching pays. The user's actual question is the only dynamic part — leave it at the end.
Don't hand-wave "use a cheap model first." Be specific: confidence from logprobs, a small classifier, length/complexity heuristics. Then a fallback path when the cheap model returns "I'm not sure."
Streaming hides latency but doesn't cut cost. Batching cuts cost but kills latency. Know which one the product needs and pick deliberately.
Agent with no max-iterations + no budget cap is how you wake up to a $5k overnight bill. Set hard caps on every loop and every per-user spend.
Don't cut cost first and check quality after. Hold quality constant via evals; cost-cut against the eval. Otherwise you're shipping regressions you can't see.
Adds: training pipeline, model storage, deployment, monitoring, retraining cadence. Worth it for high-volume narrow tasks; rarely worth it for general chat.
"$0.04 per resolved support ticket" lands better than "we use Sonnet 4.5." Reframe to per-outcome math when discussing tradeoffs.
Go deeper (optional):
Provider docs spell out the exact caching rules and discounts: Anthropic's prompt-caching guide, OpenAI's prompt-caching and Batch API pages, and each provider's pricing table (the per-million input/output numbers used above change over time — always price your own model). For the eval/observability side, the Langfuse and Braintrust docs walk through wiring traces to eval scores. None of it is required to answer the interview question — the ordering and the math here are the load-bearing part.