Retrieval-augmented generation
As of June 2026: vector DB vendors, embedding models, and API pricing cited below reflect that date — confirm before relying on them.
📖 Walk me through it — plain English
An LLM (large language model — the AI that writes text, like ChatGPT) only knows what it learned during training. It has never seen your company's docs, your codebase, or yesterday's support tickets. RAG (retrieval-augmented generation) fixes that. Instead of expecting the model to memorize your private data, you look up the relevant snippets at question-time and paste them into the prompt, then ask the model to answer using those snippets. "Retrieval" = the lookup step; "augmented generation" = the model generates its answer augmented by (helped by) what you found.
Everyday analogy: imagine an open-book exam. The student (the LLM) is smart but hasn't memorized the textbook. So before answering each question, you flip to the most relevant pages and lay them on the desk. The student then writes the answer by reading those pages. RAG is the machinery that, given a question, instantly finds the right pages out of millions and puts them in front of the model. A bad answer usually means you handed it the wrong pages — not that the student is dumb.
Why can't we just keep all the pages on the desk? Because there are too many. So we pre-process the library once: chunk each document into bite-size pieces (a few paragraphs, here 200–800 tokens — a token is roughly 3/4 of a word), then embed each chunk. Embedding turns text into a list of numbers (a "vector") that captures its meaning, so that two chunks about the same idea end up as nearby points in space, even if they use different words. We store all these vectors in a vector database (a search engine built to answer "which stored vectors are closest to this one?").
Here is the whole pipeline as one slow walk-through. Follow a single question through it:
Why this design works: the hard part of "chat with your docs" is almost never the model writing prose — it's putting the right pages on the desk. That's why the lesson hammers chunking (cut on natural boundaries like headings, with 10–15% overlap so a sentence isn't split mid-thought), hybrid retrieval (meaning-search plus keyword-search cover each other's blind spots), reranking (cheap wide net, then a precise filter), and citations (so a human can verify, turning the product from a confident oracle into a checkable researcher). In an interview, naming this full pipeline — and especially evaluating retrieval on its own — is what signals you've actually shipped one.
Table-stakes for any "chat with your docs / data / codebase" product. Perplexity, Harvey, Glean, Sierra, and any internal-AI-search team grade this. The interview question is usually "design a RAG system for X" — they want the whole pipeline, not just "use embeddings."
- Ingest — parse documents (PDFs, HTML, code).
- Chunk — split into 200–800 token pieces with overlap. Strategy matters (semantic vs fixed-size).
- Embed — vectorize chunks (OpenAI text-embedding-3, Cohere, BGE).
- Index — store in vector DB (Pinecone, Weaviate, pgvector, Qdrant, Chroma).
- Retrieve — embed query, top-k similarity; often hybrid with sparse (BM25).
- Rerank — cross-encoder (Cohere Rerank, BGE-reranker) sharpens top-k → top-n.
- Assemble prompt — system + retrieved chunks + user question; cite sources.
- Generate — LLM produces grounded answer.
- Evaluate — answer faithfulness, retrieval recall, end-to-end accuracy.
Fixed 512-token chunks lose semantic boundaries. Semantic chunking (split on headings, paragraphs) + 10–15% overlap is the modern default.
Vector search misses exact-keyword matches (product codes, names). Combine BM25 + vector with reciprocal rank fusion (RRF). 10–20% recall lift, very cheap.
Top-50 from retrieval → cross-encoder rerank → top-5. Bi-encoder retrieval is fast but imprecise; cross-encoder is precise but slow. Two-stage gets both.
Always cite source chunks in the answer. Without citations users can't verify; the product feels like an oracle (bad) instead of a researcher (good).
Top 3: (1) right doc not retrieved, (2) wrong doc retrieved confidently, (3) model ignores the context and hallucinates. Eval each separately.
Answer-quality evals hide retrieval bugs. Measure recall@k on a golden set of (query, expected_chunks) pairs.
Re-embedding unchanged docs on every deploy = wasted money. Hash-cache by content.
| DB | Best for | Watch out for |
|---|---|---|
| pgvector | Already-Postgres stacks, <10M vectors, transactional joins with metadata. Default pick for most product apps. | HNSW index build memory; performance falls off above ~10M without partitioning. |
| Pinecone | Managed, scales to billions, multi-tenant namespaces, low-ops. Enterprise-default for "vector search as a service." | Cost at scale; vendor lock-in; opinionated APIs. |
| Weaviate | Self-host or managed, built-in hybrid (BM25+vector), strong filtering. Good for "I want hybrid out of the box." | Ops burden if self-hosting; smaller community than Postgres. |
| Qdrant | Rust-fast, payload filtering, hybrid, runs on a laptop or scales out. Good Rust/perf-conscious pick. | Smaller ecosystem; less third-party tooling than pgvector or Pinecone. |
| Chroma | Embedded / local dev, prototypes, single-process apps. Zero-ops. | Not for production scale; treat as the "SQLite of vector DBs." |
| Turbopuffer | Object-storage-backed, very cheap at scale (billions of vectors for cents/GB). Newer; used at Cursor, Notion. | Latency floor higher than in-memory; eventually consistent. |
| Elasticsearch / OpenSearch | You already have it for search + want dense vectors alongside BM25 in one query. | Vector recall trails dedicated vector DBs; tuning is harder. |
The senior move: name the scale + operational constraint first, then pick. "<10M vectors and we're already on Postgres → pgvector. Billions of vectors, multi-tenant → Pinecone or Turbopuffer." Anyone naming a vector DB without naming the constraint is guessing.