Token budgets, caching, compaction

Let's define the words first. A large language model (LLM) is the AI behind tools like ChatGPT or Claude. Every time you talk to one, you send it text and it sends text back. The model doesn't read words — it reads tokens, which are chunks of text roughly the size of a syllable or short word (about 0.75 tokens per English word). The context window is the total pile of tokens the model can hold in its "working memory" at once — today that's around 200,000 tokens, which sounds enormous but fills up fast in a long conversation.

Here is the single most important fact to internalize, the on-ramp for everything below: the model only knows what is in its context window right now. It has no memory of your last conversation, no live access to your database, no idea what your code looks like — unless those things are sitting in the tokens you sent this turn. The window is a fresh desk wiped clean before every call, and you re-place onto it exactly the papers the model needs. So your job is curation: decide what to put on the desk, in what order, and what to leave off. That curation discipline has a name — context engineering: the practical craft of deciding what text to feed the model, in what order, so the result is cheap, fast, and accurate.

People often confuse this with prompt engineering, so let's separate them. Prompt engineering is about wording a single instruction well — phrasing the question, giving a role ("you are a careful editor"), asking for a specific format. Context engineering is the bigger discipline of assembling the whole pile of tokens around that instruction: which documents, which past turns, which examples, which tool outputs, and how they're ordered and trimmed. Prompt engineering is one ingredient; context engineering is the whole plate. It's "unglamorous" because it isn't a clever algorithm — it's the production discipline that turns a flashy demo into something you can actually afford to run for thousands of users. Anthropic asks about it directly for AI-product roles because two of the biggest levers here — prompt caching and compaction — are things they invented.

A quick tour of the parts of the pile, since the rest of the lesson uses these words. A system prompt is the standing instructions you set once — the model's role, rules, and tools; it usually comes first and rarely changes. The user prompt is the actual request for this turn — it changes every call. Tool results are text the model gets back after it calls a function you gave it (a database row, an API response, a file's contents); those land back in the context window and count toward the budget like everything else. Memory is any durable note you choose to carry forward across turns or sessions — a user's name, a decision made earlier — which you must explicitly re-inject, because the model won't remember it on its own.

Here's the everyday analogy. Imagine you're a lawyer briefing a busy expert who bills by the minute. Every page you hand them costs money, and the expert can only truly focus on so many pages before their attention gets fuzzy. So you don't dump the entire case file on their desk every meeting. You keep a fixed cover sheet they've already memorized (cheap to "re-read"), you hand over only the two pages relevant to today's question (not the whole 50-page contract), and when the meeting log gets too long you replace old back-and-forth with a one-paragraph summary. That is exactly what these techniques do for an LLM.

The four big moves the lesson lists map cleanly onto that:

• Cache the stable prefix. Prompt caching means "save the work of reading a chunk of tokens so you don't pay to re-read it next call." If the unchanging part of your prompt — the system instructions, the tool definitions, the long background doc — always comes first, the model can reuse that prefix from cache at roughly 10% of the normal cost. Only the new bit at the end (the user's actual question) gets charged full price. Order matters: stable stuff first, then the changing user input last. (Caching keys on an exact prefix match, so even one edited word near the top busts the cache for everything after it.)
• Token-budget your context. Treat tokens like a spending limit. More context is not automatically better — past a point the extra text is noise that drowns out the signal and actually makes answers worse. So you deliberately decide how many tokens each part of the prompt is allowed.
• Compaction. In a long agent session the conversation eventually outgrows the window. Compaction (also called summarization) = replace the old turns with a short summary of them while keeping the most recent turns word-for-word. It's lossy (you throw away detail) but necessary, like replacing old meeting minutes with a recap.
• Don't paste, retrieve. Instead of jamming a 50,000-token document into every single turn, store it elsewhere and fetch only the ~2,000 tokens that are relevant right now. That fetch-the-relevant-bit pattern is called RAG (retrieval-augmented generation): you retrieve matching snippets from a store and add them to the context so the model can generate an answer grounded in them. Getting good snippets depends on chunking — splitting big documents into bite-sized pieces ahead of time so you can fetch just the relevant ones instead of the whole file.

One more tool that quietly shapes context: few-shot examples. Instead of describing the format you want, you paste two or three worked input→output pairs right into the prompt; the model imitates the pattern. It's the cheapest way to steer behavior — but each example spends tokens, so it's a budget decision like everything else.

Two more facts worth fixing in your head. Time-to-first-token — how long before the model starts replying — grows roughly in step with how much input you send, because the model must "read" (prefill) all of it first; 100K tokens of input can mean a 5–10 second wait. And context rot: as a context window fills up, the model's overall ability to use it reliably degrades — a close cousin of the lost-in-the-middle effect, where facts placed in the middle of a long context are recalled worse than facts at the very start or very end. So put the important stuff early (or repeat it right next to the question), and keep stale context — outdated docs, resolved sub-tasks, dead tool output — out of the pile entirely.

Why does this matter so much? Because output tokens cost 3–5× more than input, and cached input costs about a tenth of uncached. Put those together and a thoughtful prompt layout can be a 10× difference in your bill per call — real money at scale. That's the "interview tell" at the bottom: when someone asks "how would you make this LLM feature cheaper," a weak answer is just "use a cheaper model." A strong answer names the levers in this lesson — prompt caching, retrieving instead of pasting, compacting long histories, and using small models for the easy sub-tasks.