AI infrastructure in a normal design round
You do not have to be an ML engineer for an ML question to land in your design round. Because nearly every product now has a model somewhere in it, general SWE system-design interviews increasingly bolt an AI component onto a normal problem: "add a recommendation feed", "add semantic search", "add an LLM support assistant." This lesson gives you the small, durable vocabulary to handle that calmly — the infra around the model, which is exactly the part a software engineer owns.
📖 Walk me through it — plain English
The key reframe: a model is just a slow, expensive, sometimes-wrong remote service. Once you see it that way, everything you already know about system design applies. A model call is like a call to a third-party API that costs real money per request, takes hundreds of milliseconds to seconds to answer, occasionally returns garbage, and sometimes is completely down. Your job as the surrounding engineer is to make a reliable product on top of that unreliable, costly box.
That means the same tools as always, pointed at a new target. Caching, because the same question gets asked a lot and each answer costs money. A latency budget, because the model is your slowest hop and users won't wait. Versioning and A/B testing, because the "code" (the model or the prompt) changes often and you must be able to compare a new version against the old and roll back. Graceful degradation, because GPUs run out and providers have outages, and "the AI feature is down" shouldn't take the whole page down.
You're not designing the neural network. You're designing the queue in front of it, the cache beside it, the fallback behind it, and the dashboard watching it. That's pure software engineering.
The vocabulary, defined once
- Inference: running a trained model to get an output for one input (a prediction, an embedding, a generated answer). The per-request work you're building around. "Training" is the offline, one-time-ish job that produced the model; "inference" is the online, every-request cost.
- Latency budget: the total time you'll allow a request, divided across its hops. If the page must respond in 500ms and the model alone takes 400ms, you have 100ms for everything else — which forces choices (smaller model, cache, async).
- Embedding: a list of numbers that represents the meaning of an item (text, image), so that similar things sit close together in that number-space. The basis of semantic search and recommendations.
- Vector database: a store specialized for "find the items whose embeddings are nearest to this one" (approximate nearest-neighbor search). It's how you retrieve by meaning instead of exact keywords.
- Model versioning: treating each model (and each prompt) as a deployable artifact with a version, so you can roll forward, roll back, and know exactly which version produced a given output.
- Inference-layer A/B test: routing some traffic to model v2 and the rest to v1, then comparing a business metric (click-through, resolution rate), because offline accuracy doesn't always predict real-world win.
- Graceful degradation: a defined fallback when the model is slow, over budget, or down — a cheaper model, a cached/heuristic answer, or hiding the feature — so the rest of the product survives.
Worked example: "add semantic search to our product catalog"
Watch how the normal design framework absorbs an AI feature. The ask: users type "warm jacket for hiking" and get relevant products even if those exact words aren't in the listing.
Step 1 · Split offline vs online. Offline (a batch job): run every product through an embedding model, store each product's embedding in a vector DB. This is precomputed, so it's off the request path. Online (per query): embed the user's query, ask the vector DB for the nearest product embeddings, return those. The expensive embedding of millions of products happens once, not per search.
Step 2 · Latency budget. Search must feel instant — say 200ms. Embedding the short query is fast (~20ms); the vector DB nearest-neighbor lookup is the main cost — pick an approximate index (HNSW/IVF) that trades a sliver of recall for big speed. Cache embeddings for popular queries so repeat searches skip the model entirely.
Step 3 · Keep the model fresh and versioned. When you swap the embedding model, the new query embeddings live in a different number-space than the old product embeddings — so you must re-embed the whole catalog with the new model before switching. Version both; A/B the new index against the old on a real metric (add-to-cart rate), and keep the old index warm so rollback is instant.
Step 4 · Degrade gracefully. If the embedding service is down or over budget, fall back to plain keyword search. The user gets a slightly worse result, not an error page. Name this in the round — it's the operational reflex from the previous lesson, applied to the model.
Notice every step was ordinary system design — batch vs online, budgets, versioning, fallback. The "AI" was contained to one box you wrapped with familiar tools.
The LLM-specific twists
When the model is a large language model (an LLM — a model that generates text token by token), three extra concerns show up:
- Streaming responses: generation is slow, so you stream tokens as they're produced (server-sent events) instead of waiting for the whole answer — perceived latency drops even though total time doesn't.
- Semantic caching: exact-match caching barely helps when every prompt is slightly different, so you cache by embedding similarity — "have we answered a question very close to this one?" — to reuse costly generations.
- Cost per token + context limits: you pay per token in and out, and the model has a finite context window, so trimming and summarizing what you send is both a cost and a correctness lever.
These three (plus retrieval to ground answers in your own data) are the heart of the two full LLM case studies later in the guide. Here, just be able to name them when an LLM appears mid-round.
Pitfalls
Embedding or scoring millions of items per query. Precompute offline and store; do only the tiny per-request piece online.
GPUs run out and providers have outages. No fallback means your AI feature can take the whole page down. Always name a degradation path.
A new embedding model's vectors aren't comparable to the old ones. You must re-embed the corpus before switching, or search quality collapses.
Model calls cost real money. With no cache and no budget, a viral feature becomes a runaway bill. Cache hot answers and cap spend.
Takeaway: treat a model as a slow, costly, sometimes-down remote service and the rest is ordinary system design. Precompute embeddings offline and keep only the small piece online; set a latency budget; version the model and prompt and A/B on a real metric; cache hot answers; and always name a graceful-degradation fallback. For LLMs, add streaming, semantic caching, and cost-per-token/context awareness. The model is one box — your job is the infra around it.
→ Going deeper: the two full LLM design walkthroughs build directly on this — AI customer-support copilot and code-review agent — and LLM economics covers cost-per-token in depth. The fallback thinking comes straight from the operational layer.