Designing an LLM agent

As of June 2026: framework names, tool APIs, and orchestration patterns cited below reflect that date — confirm before relying on them.

Standard at AI labs (Anthropic Applied AI, OpenAI), agent-product startups (Sierra, Decagon, Cognition, Adept), and any team building "AI does multi-step tasks." Whiteboard prompt is usually "design an agent that does X" — support analyzer, research assistant, ticket triager, deploy bot. Like system design, there's no single right answer; they grade the decomposition.

The vocabulary, defined once

This round is full of jargon that sounds harder than it is. Every term below names one small, concrete idea. Read these once and the rest of the lesson clicks into place.

• LLM agent — a large language model wrapped in a program that lets it act in a loop: call a tool, read the result, decide the next move, repeat, until it produces a final answer. Plain answering ("here's a poem") is one shot; an agent is many shots aimed at finishing a task.
• Tool / function calling — a "tool" is just a normal function you wrote (get_weather(city), refund(order_id)). "Function calling" is the model feature that lets the LLM request that one of those functions be run, with arguments it chooses. The model does not run the function itself; it emits a structured request like {"tool":"get_weather","args":{"city":"NYC"}}, your code runs the real function, and you hand the result back.
• Tool schema — the machine-readable description of a tool: its name, what it does, and the type of each argument (e.g. order_id is a string). You give the model these schemas so it knows what buttons exist and how to fill them in. The schema is where you encode constraints — a good schema is worth a paragraph of prompt instructions.
• The agent loop — the outer while loop your code runs: send the conversation so far to the model, get back either a tool call or a final answer; if it's a tool call, run it, append the result, and loop again; if it's a final answer, stop. That loop is the agent.
• ReAct (the observe–think–act loop) — "Reason + Act." Each turn the model writes a short thought (think), then issues a tool call (act), then reads the returned observation (observe) before the next thought. It interleaves reasoning and action one step at a time, with no master plan written up front.
• Planning — having the model write out a multi-step plan before acting ("1. find the order, 2. check the policy, 3. refund or escalate"), then execute the steps. Helps on long tasks where reacting blindly would wander; overkill for short ones.
• Memory / scratchpad — where the agent keeps state. The scratchpad is the running record of thoughts, tool calls, and observations for the current task (it lives in the model's context window). Long-term memory is anything you persist across tasks — past preferences, prior tickets — usually stored in a database and pulled back in when relevant.
• Guardrails — the checks that keep the agent inside safe, correct behavior: input filters, limits baked into tools, output filters, and approval gates. Guardrails are code, not vibes.
• Termination condition — the rule(s) that end the loop: the model emitted a final answer, OR it hit the max-iteration cap, OR it blew the cost/time budget, OR it called an exit tool like escalate_to_human. Every agent needs at least one, or it never stops.
• Human-in-the-loop — pausing the agent to get a person's approval before an action runs (refund, delete, deploy, public post). The model proposes; a human commits the irreversible step.
• Multi-agent — splitting a big job across several agents: an orchestrator agent dispatches specialist sub-agents (one researches, one writes code, one summarizes) and combines their results. Useful when sub-problems are independent or would otherwise pollute one shared context.
• Agent vs. fixed workflow — a fixed workflow is hard-coded steps you wrote in order (do A, then B, then C); the path is decided by you at build time. An agent lets the model decide the path at run time. Use a fixed workflow when the steps are known and stable (it's cheaper, faster, debuggable); reach for an agent only when the right next step genuinely depends on what was found mid-task.

The agent loop, in actual code

The loop sounds abstract until you see it. Here is the entire skeleton — twenty lines that turn a one-shot model into an agent. Read the comments; this is the whole game.

# The agent loop: ask the model, run any tool it requests, repeat until done.
messages = [{"role": "user", "content": task}]   # the scratchpad starts here
for step in range(MAX_STEPS):                # the LEASH: termination by iteration cap
    reply = model.call(messages, tools=TOOL_SCHEMAS)  # brain decides next move

    if reply.is_final_answer:               # TERMINATION: model says it's done
        return reply.text

    # Otherwise the model asked to call a tool. Run it (the "hands").
    tool_name, args = reply.tool_call
    try:
        observation = TOOLS[tool_name](**args)   # run YOUR real function
    except Exception as e:
        observation = f"ERROR: {e}"          # feed errors back, don't crash

    messages.append(reply.tool_call)            # record the act...
    messages.append({"role": "tool", "content": observation})  # ...and the observation

return escalate_to_human("hit step limit without finishing")  # safe fallback

Notice four things baked in. (1) MAX_STEPS is the leash — the loop physically cannot run forever. (2) The try/except turns a tool crash into a normal observation the model can react to, instead of blowing up the program. (3) Each turn appends to messages — that growing list is the scratchpad. (4) The fallback after the loop means even "ran out of steps" lands in a defined, safe place rather than silence.

A worked example: the ReAct loop, step by step

Now watch the model think. Task: "Customer #4471 says their order never arrived and wants a refund." The agent has four tools: get_order(order_id), get_shipping_status(order_id), refund(order_id), and escalate_to_human(reason). Each turn is a thought (reason), a tool call (act), and an observation (the result it reads back). This trace is exactly what shows up on the scratchpad:

# --- Step 1 ---
Thought: I need to find this customer's most recent order before I can act.
Action:  get_order(order_id="4471")
Observation: { id: 4471, item: "headphones", total: 80, status: "shipped" }

# --- Step 2 ---
Thought: It shipped. Did it actually get delivered? Check tracking before refunding.
Action:  get_shipping_status(order_id="4471")
Observation: { carrier: "UPS", state: "lost_in_transit", delivered: false }

# --- Step 3 ---
Thought: Confirmed lost, not delivered. Total is $80, under the $500 auto-refund cap,
         so I'm allowed to refund without a human.
Action:  refund(order_id="4471")
Observation: { ok: true, refunded: 80, confirmation: "RF-9920" }

# --- Step 4 ---
Thought: Refund succeeded. I have everything I need; time to answer the customer.
Final answer: "Your headphones were lost in transit by UPS, so I've refunded the
                full $80 (confirmation RF-9920). Sorry for the trouble!"

Trace what made this work. The model chained tools — it didn't refund blindly; it verified the order existed, then verified the package was actually lost, then checked the amount against a policy it knew, and only then acted. The observation from each step fed the next thought. And it terminated on its own by emitting a final answer once the goal was met. Now imagine the order total were $900: at step 3 the model should reason "over the $500 cap" and call escalate_to_human("refund $900 > cap") instead — a different, equally valid termination. That branch is exactly what the tool-side guardrail enforces even if the model forgets.

The key design decisions (say these out loud)

Interviewers want to hear you weigh tradeoffs, not recite features. Four decisions carry most of the signal:

• The tool set. Pick ~5–15 tools, each doing one clear thing with a tight schema. Decide what not to expose: every tool the model can call is attack surface and a chance to pick wrong. Prefer read-only tools where possible; gate write/destructive tools.
• Stopping (termination). Spell out every way the loop ends: success (final answer), give-up (escalate tool), and the hard caps (max iterations + cost/time budget). "What stops it?" with no answer is an instant red flag.
• Error handling. A tool will return an error, time out, or hand back garbage. Decide: feed the error back so the model can retry or try another tool (good for transient failures), cap retries so a flaky tool can't spin the loop, and escalate after N failures rather than guessing.
• Cost / latency. Each model hop is ~1–5s and real token spend. State the budget per task, prefer the cheapest model that passes eval, cache or batch where you can, and replace any step that can be deterministic code with deterministic code. A ten-hop agent is ~10× a single call — say that number.

Pitfalls (name the fix, not just the failure)

Listing failure modes is table stakes; pairing each with its mitigation is what reads as senior. The four that come up every time:

• Infinite / wasted loops. The model keeps calling tools without converging (often re-calling the same tool with the same args). Fix: a max-iteration cap, a cost/time budget, loop-detection (bail if the last two steps repeat), and an escape-hatch tool so it can give up cleanly.
• Hallucinated tool arguments. The model invents an order_id it never saw, or passes the wrong type. Fix: validate arguments against the schema before running, reject and return a clear error the model can correct from, and design tools that fail loudly on bad input rather than doing something plausible.
• No guardrails (the dangerous one). A tool blindly executes whatever the model asks — including a $10,000 refund or a delete triggered by a prompt injection. Fix: put the limit inside the tool (cap, confirmation token, allow-list), gate irreversible actions behind human approval, and treat model output as untrusted input.
• The model gives up or bluffs. Without an escape hatch, a stuck model emits a confident, wrong answer. Fix: always provide escalate_to_human / request_clarification, and reward "I'm not sure, escalating" over a fabricated resolution in your eval.