Token streaming, partial render, cancellation

As of June 2026: streaming API shapes and SDK patterns cited below reflect that date — confirm before relying on them.

Surfaces specifically at Cursor, Vercel (v0), Perplexity, Linear AI, any "chat UI for an LLM" product. The interview problem: "build a streaming chat component" or "this LLM call freezes the UI for 8 seconds — fix it." Tests whether you understand SSE/WebSocket streaming, React render performance under high-frequency updates, and graceful cancellation.

Why stream at all? Perceived latency

Before any of the machinery, the why. A long LLM answer might take 8 seconds to finish completely. You have two choices: wait for all 8 seconds and dump the whole answer at once, or show the first words after roughly half a second and let the rest trickle in. Both finish at the same wall-clock moment — but they feel wildly different. The thing that matters to a human is not when the answer finishes; it's when something starts happening. That gap — request sent until the first visible token — has a name: TTFT, "time-to-first-token." Streaming exists almost entirely to shrink the felt wait, which we call perceived latency (how slow it feels, as opposed to how slow it measurably is).

A spinner that sits for 8 seconds reads as "broken." Text that begins appearing in 500ms reads as "working, and fast," even when the total time is identical. The same trick of showing progress immediately — before the real result is confirmed — is sometimes called optimistic UI (you optimistically render the in-progress state rather than waiting for the final, fully-confirmed one). Streaming is the LLM-shaped version of that idea: every token is a little proof that the system is alive and making progress.

The vocabulary, defined once

Every term you'll hear in this problem, in plain English, so nothing below is a mystery word:

• Token streaming — sending the model's output piece by piece (token by token) as it's generated, instead of one big response at the end.
• Server-Sent Events (SSE) — a built-in browser feature for a one-way push channel: the server holds an HTTP response open and keeps writing little text events down it ("here's a token… here's another…") until it's done. One direction only: server → client.
• WebSocket — a persistent two-way connection: both sides can send at any time. More powerful, more wiring. Overkill when only the server needs to talk.
• Chunked transfer — the HTTP mechanism underneath that lets a server send a response in pieces ("chunks") without knowing the total size up front. It's how the connection stays open and dribbles data out; SSE rides on top of it.
• TTFT (time-to-first-token) — milliseconds from "request sent" to "first token shown." The metric streaming is built to minimize.
• Backpressure — what happens when the producer (server, sending tokens fast) outpaces the consumer (a slow client or slow render). Pressure builds up in the buffer; you must relax it by slowing down, coalescing, or dropping intermediate state.
• Optimistic UI — showing the expected/in-progress result immediately rather than waiting for full confirmation, to make the app feel instant.
• Partial rendering — drawing the answer while it is still incomplete (a half-written paragraph), then updating in place as more arrives.
• Markdown-during-stream — the problem of formatting text (bold, lists, code blocks) that isn't finished yet, where syntax characters are only "half typed."
• Cancellation / AbortController — stopping a request in flight. AbortController is the browser object that carries a "cancel" signal you hand to fetch.
• Retries — trying again after a failure. For streams, a retry means re-running the whole request, not resuming from where it broke.
• Idempotency — a property where doing the same operation twice has the same effect as doing it once. It matters because retries can cause double-work (two answers, double billing) unless the request is safely repeatable.

What an SSE stream actually looks like on the wire

SSE is not magic — it's just plain text in a kept-open HTTP response. The server sets the content type to text/event-stream and writes blocks separated by blank lines. Each block has a data: line. Here's a tiny slice of what the browser receives as the model produces "Hello there":

// Response header tells the browser "this is a live event stream":
//   Content-Type: text/event-stream

data: {"token": "Hello"}

data: {"token": " there"}

data: [DONE]

Two things to notice. First, each event is its own little JSON payload — you parse them one at a time. Second, there's a [DONE] sentinel at the end so the client knows the model finished cleanly (versus the connection just dying). That distinction — "ended on purpose" vs "the line went dead" — is exactly what your error handling later hinges on.

Consuming the stream on the client, step by step

Here is the smallest honest example of reading a stream in the browser and appending tokens as they arrive. The pattern is always: open a fetch, grab a reader off the response body, and loop — each loop turn hands you a chunk of bytes, which you decode to text and append. (A "reader" is just the object that lets you pull the next piece of an incoming stream.)

// 1. Make a controller so we can cancel later (see Cancellation section).
const controller = new AbortController();

const res = await fetch("/api/chat", {
  method: "POST",
  body: JSON.stringify({ prompt }),
  signal: controller.signal,   // hand the cancel signal to fetch
});

// 2. res.body is a stream of bytes. Get a reader and a text decoder.
const reader  = res.body.getReader();
const decoder = new TextDecoder();
let text = "";

// 3. Loop until the stream says it is done.
while (true) {
  const { value, done } = await reader.read();
  if (done) break;                         // no more bytes coming
  const chunk = decoder.decode(value);   // bytes -> text
  text += chunk;                            // append the new piece
  scheduleRender(text);                     // batched draw, see next section
}

Note one subtlety glossed over for clarity: a single read() chunk may not line up with the SSE event boundaries — you might get half of one data: line. Real code keeps a small leftover buffer and only parses up to the last complete blank-line separator, stashing the partial remainder for the next loop. That same "wait for a clean boundary before parsing" instinct is exactly what saves you on markdown below.

Batching the render so it doesn't thrash

The call to scheduleRender above is doing real work. If you instead drew on every single chunk, you'd ask the browser to re-layout the message 50–200 times a second and the UI would stutter. The fix is to coalesce: remember the latest text, and schedule exactly one paint per animation frame (about every 16ms at 60fps). requestAnimationFrame is the browser's "call me right before the next repaint" hook — perfect for this:

let pending = null;
let scheduled = false;

function scheduleRender(latestText) {
  pending = latestText;                // always keep the newest, drop the stale
  if (scheduled) return;            // already a paint queued? do nothing
  scheduled = true;
  requestAnimationFrame(() => {
    setMessage(pending);              // one state update per frame, max
    scheduled = false;
  });
}

This is also your first taste of handling backpressure: if tokens flood in faster than the screen can repaint, we deliberately throw away the intermediate snapshots (we only keep pending, the latest) and never the final one. The user never needed to see the in-between states anyway — only smooth, current text. Coalescing-and-dropping intermediate state is the standard relief valve when a producer outruns a consumer.

Cancellation, both sides

The browser half is small: keep the AbortController from the fetch, and call abort() when the user clicks Stop or fires a new question. The read() loop will then throw an AbortError, which you catch and treat as "stopped on purpose," not "crashed."

// Stop button handler:
stopButton.onclick = () => controller.abort();

// New-question handler: cancel the old one first, then start fresh.
function ask(prompt) {
  if (controller) controller.abort();   // latest request wins
  controller = new AbortController();
  // ...start the fetch from the previous section with this signal...
}

But aborting the fetch only hangs up your end. The server may still be paying for the model to generate tokens into the void. The server-side half: when the LLM SDK is given the request, pass it an abort signal too, and wire the HTTP request's "client closed the connection" event to fire it. Concretely, on the backend you listen for the request being aborted and then abort the upstream model call — so the moment the browser hangs up, the expensive generation stops and the meter stops with it. Saying "cancellation is two-sided" out loud, and explaining the billing reason, is the single highest-signal sentence you can offer on this question.

Errors, retries, and idempotency

Streams fail partway more often than normal requests, because the connection is held open for seconds. When the line dies mid-answer, the right UX is: keep whatever text already arrived on screen, change the status from "loading" to "stopped" (a spinner that never resolves is the worst outcome), and offer a Retry. Crucially, a retry of an LLM stream is a full re-run — you can't resume from token 200, because the model has no memory of the half-answer; you send the prompt again and get a fresh generation.

That re-run is where idempotency earns its keep. If the user (or an automatic retry) fires the same request twice, you don't want two charged generations or two duplicated messages appended to the thread. The common guard is an idempotency key — a unique id you attach to the request so the server can recognize "I've already seen this one" and return the in-flight or cached result instead of starting a second expensive call. The interview-level summary: retries are necessary because streams are fragile; idempotency is what keeps retries from doing damage.

Markdown while it's still being typed

LLMs emit markdown — **bold**, lists, fenced code blocks. The hazard is that mid-stream the syntax is half-finished. If the model has emitted ```python to open a code block but hasn't yet emitted the closing ```, a naive markdown renderer sees an unclosed fence and may format the entire rest of the document as code — or, with an unmatched *, smear everything into bold-italic. The defense mirrors the SSE-boundary trick: don't fully re-parse on every keystroke of input. Either parse with a renderer that tolerates incomplete input (treating an open fence as "code so far"), or defer the heavy markdown pass to chunk boundaries / stream end and show plain text in between. The principle is the same one twice: only commit to an interpretation once you have a clean boundary.

UX concerns and pitfalls

• Stop button. Always offer one during a stream, and make it actually cancel both sides. A Stop that only blanks the UI but keeps the server generating is a billing leak, not a feature.
• Error mid-stream. Render what arrived, switch status to "stopped," show Retry. Never leave a spinner spinning forever — distinguish "finished" (saw [DONE]) from "died."
• Scroll behavior. Auto-scroll to keep the newest text in view, but stop auto-scrolling the instant the user scrolls up to read earlier output. Yanking them back to the bottom on every token is one of the most hated streaming bugs; the usual rule is "stick to bottom only while the user is already at the bottom."
• Tool calls mid-stream. Modern APIs interleave tool_use blocks in the stream ("thought → calls a tool → tool result → more thought"). The layout must absorb these inline panels without jumping or breaking, and your parser must not mistake a tool block for answer text.
• The latest request wins. If a user fires a second question before the first finishes, abort the first; otherwise two streams append to the same box and interleave into nonsense.
• Don't setState per token. The headline pitfall — it looks fine on a one-sentence reply and falls apart on a long answer. Batch from the start.