A large language model generates responses one token at a time. A 600-token response might take 8 to 10 seconds to complete, even though the first token may be ready in less than a second. With streaming, users can start reading as tokens arrive instead of waiting for the entire response.

Streaming text to a browser is straightforward, but production systems introduce additional challenges. Connections can drop midway through a response, users can click "Stop" while the model continues generating and consuming tokens, and tool calls may arrive as incomplete JSON. Realtime voice adds even tighter constraints. The assistant needs to respond quickly, handle interruptions, and stop speaking when the user starts talking.

In this chapter, we'll cover how to stream LLM responses, handle disconnects and cancellation, process structured output and tool calls, move long-running tasks to background jobs, and design realtime voice systems that meet strict latency requirements.

For most web requests, latency is measured by how long it takes to return the complete response. With LLMs, two metrics matter for the user experience:

Without streaming, users wait for TTFT plus the full generation time before seeing a response. With streaming, they can start reading as soon as tokens arrive while the model generates the rest. The total generation time and compute cost remain largely unchanged, but the response feels much faster.

This makes streaming the default choice for interactive GenAI applications. It also changes what we optimize. Once responses are streamed, shortening the wait before the first token often improves the user experience more than speeding up the end of the response, because the user is already reading it.

However, streaming isn't always necessary. If the output is consumed by another program, such as a classification label, an extracted JSON record, or a summary stored in a database, there's little benefit to receiving it token by token. A standard request-response approach is simpler to implement, validate, and retry.

Most LLM streaming applications use one of two transports: Server-Sent Events (SSE) or WebSockets. The main difference is whether data needs to flow in one direction or both.

Server-Sent Events

Server-Sent Events (SSE) is a lightweight streaming mechanism built on HTTP. The client sends a request, and the server responds with the content type text/event-stream. Instead of closing the response, the server keeps the connection open and sends events as they become available.

Each event consists of a few lines of text followed by a blank line:

The id field identifies the event, event specifies its type, and data contains the payload. Lines starting with a colon are comments, often used as heartbeats to keep idle connections alive.

SSE works well for text chat because communication is mostly one-way. The user sends a message, and the server streams back the response. It works with standard HTTP infrastructure and is widely used by LLM providers for streaming APIs.

The browser's built-in EventSource API automatically reconnects when a connection drops and includes the last received event ID in the Last-Event-ID header. The server can use this ID to resume the stream if it retains previous events.

However, EventSource only supports GET requests and doesn't allow custom request headers or a request body. Most chat applications send messages using POST requests with authentication headers, so they typically consume SSE streams through fetch() and implement reconnection themselves.

WebSockets

WebSockets provide a persistent, full-duplex connection that allows the client and server to send messages independently at any time.

This becomes useful when the client needs to send data while the server is responding. For example, a voice assistant may continuously send microphone audio while receiving generated speech. A collaborative editor may send user changes while receiving AI suggestions. With SSE, sending additional client messages requires separate HTTP requests.

The trade-off is additional operational complexity. WebSocket connections are long-lived and tied to a server instance, so deployments need connection draining, and reconnecting clients may need their session state restored. Applications also need to handle heartbeats, reconnection, and application-level message formats.

Choosing Between Them

For most applications, use SSE for text streaming and WebSockets when continuous bidirectional communication is required. A "Stop" button alone doesn't justify WebSockets, since cancellation can be handled through a separate HTTP request.

Infrastructure That Breaks Streams

Streaming issues in production often come from the infrastructure between the application and the client rather than the streaming code itself.

Streaming raw text is enough for a demo, but production applications need to send more than just generated tokens. The client must be able to distinguish between text, progress updates, tool calls, citations, and errors.

A simple approach is to define a set of typed events:

Typed events allow the client to handle each type appropriately. For example, status updates can show what the system is doing while retrieval or tool execution is in progress. Citations can be linked to the relevant text, and errors can be displayed without losing the response already received.

Two additional details are important:

These conventions make streaming more reliable and give the client enough information to handle interruptions, retries, and partial responses correctly.

Connections drop for ordinary reasons, such as a phone switching from Wi-Fi to cellular, a laptop lid closing, or a user refreshing the page. Each disconnect raises three questions: can the user get the rest of the answer, should the model keep generating, and who pays for the tokens already produced?

Decoupling Generation From the Connection

In the simplest design, the server that calls the model also writes tokens directly to the client's connection. If the connection drops, the response is lost, and the only option is to regenerate it from scratch.

The fix is to decouple generation from delivery. A generation worker calls the model and appends every event to a durable, ordered log keyed by response ID. Redis Streams works well for this, with a TTL of a few minutes after completion. The SSE handler doesn't call the model at all. It reads from that log and forwards events to the client.

When the client reconnects, it sends the response ID and the last sequence number it received. The handler replays everything after that point from the log and then continues tailing new events. The user sees the answer pick up where it left off, and the model was never interrupted.

Because the handler only reads from a log, any server can serve the reconnect, so you don't need sticky sessions. The same mechanism also lets a user open the conversation on another device and watch the response finish there.

Cancelling Upstream Generation

A disconnect and a cancellation are not the same thing, and the system should treat them differently.

When the user clicks "Stop," they've explicitly said they don't want the rest. The client calls a cancel endpoint with the response ID, and the generation worker aborts the upstream model request. For hosted APIs, closing the streaming HTTP connection is how you signal this. For self-hosted engines such as vLLM, aborting the request frees its slot in the batch and its KV cache memory, which matters for throughput.

A plain disconnect is ambiguous. The user might be gone for good or back in two seconds. A common policy is to keep generating for a short grace period, say 30 seconds, and cancel only if no client reconnects. Shorter grace periods save tokens, while longer ones give flaky mobile connections a better chance to recover.

If the worker ignores both signals, every abandoned request runs to completion. The waste per request is small, but it adds up across millions of requests.

Billing for Partial Output

Hosted providers generally bill for the input tokens and for the output tokens generated before the request stopped, though the details vary by provider. A response cancelled after 120 tokens costs 120 output tokens, not the full answer. That's one more reason to cancel promptly.

Accounting is harder when the stream ends abnormally. Many APIs report usage only in the final chunk of the stream, and that chunk never arrives if the request is aborted. Don't let usage records go missing in that case. Count output tokens yourself as they arrive, using the model's tokenizer or a close estimate, and fall back to that count whenever the provider's usage figure is absent. Then reconcile against the provider's invoice periodically.

Whether you charge your own users for partial output is a product decision. A consumer chat product might absorb it, while an API platform that resells model access might bill for what was generated, because the cost was real.

Text is easy to stream because every prefix of a sentence is still readable. Structured output doesn't have that property. A prefix of a JSON object is not valid JSON.

Partial JSON

Suppose the model is generating a product comparison as JSON. Halfway through, the stream holds something like this:

A standard JSON parser rejects it. There are two practical approaches.

Partial parsing improves perceived latency for user-facing structured output. But any decision that depends on the data must wait for the complete, validated object. A half-received number like 29 might become 299 on the next chunk. Even when the provider supports schema-constrained decoding, which guarantees the final output parses, you still need to validate the values themselves.

Tool Calls

Model APIs stream tool calls as well. The model first emits the tool name and an ID, and then streams the arguments as fragments of a JSON string. When the model requests several tools in one turn, each fragment carries an index so you can tell which call it belongs to.

The orchestrator accumulates argument fragments per call and executes nothing until the arguments are complete and pass validation. As soon as the tool name arrives, it can send a tool_call status event to the client, so the user sees "Looking up order 4417" instead of a silent pause. If the arguments are invalid, the error goes back to the model as a tool result, giving it a chance to correct itself.

Guardrails on a Stream

Output guardrails create a direct conflict with streaming. If a moderation check must approve the full response before anyone sees it, you've given up the main benefit of streaming.

A common compromise is to check in chunks. The server holds back a small buffer, often a sentence or a few hundred characters, runs fast checks on it, and releases it to the client if it passes. This adds a little latency to each chunk but keeps the stream flowing. If a later chunk fails, the server stops the stream and sends an error event, and the client replaces the partial text with a message explaining that the response was withheld.

The trade-off is that text already shown can't be unseen. For low-risk applications, chunked checks are usually acceptable. Where a single leaked sentence is a serious problem, such as when responses might expose sensitive records, check the full response before showing anything and accept the latency.

Streaming works for responses that finish in seconds or a minute or two. It's a poor fit for work that takes much longer, like a research agent spending ten minutes reading sources, a pipeline summarizing a 300-page contract, or a job classifying a million support tickets. Holding a connection open for that long is fragile, and the user often doesn't need to watch.

Queues and Job IDs

For long jobs, the API accepts the request, places it on a queue, and immediately returns a job ID. Workers pull jobs from the queue, run them, and write results and progress to a job store.

The queue absorbs bursts that would otherwise overwhelm model rate limits, and it lets workers retry failed steps without the client knowing. For multi-step agent jobs, workers should checkpoint progress after each step, so a crashed worker resumes from the last completed step instead of starting over and paying for every model call again.

Getting Results Back

The client can learn about results in three ways.

These approaches combine well. A research agent in a web app might stream progress while the tab is open, send an email when it finishes, and let the user reload the page to fetch the final report. Webhooks need care, because deliveries fail. Sign each payload so receivers can verify it came from you, retry with backoff, and include an event ID so receivers can ignore duplicates.

Batch APIs for Offline Work

When nobody is waiting at all, provider batch APIs are worth considering. You upload a file of requests, and the provider processes them within a window, typically up to 24 hours. Both OpenAI and Anthropic offer these at roughly half the price of the standard API.

Backfilling summaries for an archive, running nightly evaluations, generating embeddings for a new index, and classifying historical tickets all fit. In each case, the work has a deadline measured in hours, so trading latency for a 50% discount is an easy decision.

Voice puts much tighter latency limits on streaming. In text chat, a two-second wait before the first token feels fine. In a spoken conversation, the gap between turns is typically around 200 milliseconds, a pattern Stivers et al. (2009) found across ten languages, and once a reply takes more than a second or so, the conversation starts to feel unnatural. Users may talk over the agent, repeat themselves, or assume the call dropped.

There are two main architectures for meeting that bar.

The Cascaded Pipeline

The traditional approach chains three models together: speech-to-text (STT) transcribes the user, an LLM writes the reply, and text-to-speech (TTS) speaks it.

The key to low latency is that every stage streams. STT produces partial transcripts while the user is still speaking. The LLM starts generating as soon as the final transcript arrives. TTS begins synthesizing the first sentence while the LLM is still writing the second, so the agent starts talking long before the full reply exists.

The cascade's strength is control. You can swap each model independently, use any LLM including one with tools and RAG, inspect the transcript at every turn, and run text guardrails on both sides. The weakness is that speech becomes text in the middle, so tone, hesitation, and emphasis are lost, and each handoff adds latency.

Speech-to-Speech Models

Speech-to-speech models, such as the models behind OpenAI's Realtime API, take audio in and produce audio out within a single model. There's no intermediate transcript to wait for, which removes handoff latency, and the model can hear tone and respond with more natural prosody.

The trade-offs run the other way. You have fewer model choices, less visibility into intermediate steps, and fewer places to insert guardrails. Audio tokens are also typically more expensive than text tokens.

Cascades remain common in customer support, where tool calls, compliance logging, and predictable behavior matter. Speech-to-speech fits conversational products where natural interaction matters more than control.

The Voice Latency Budget

The metric that matters is voice-to-voice latency, measured from when the user stops speaking to when they hear the first audio of the reply. A cascaded pipeline might target something under one second, for example 800 milliseconds.

These figures are illustrative, but they show where time goes. Turn detection is often the largest single item, and model choice matters more than in text chat, since a slow first token directly delays speech.

Tool calls are hard to fit into this budget. A lookup that takes 1.5 seconds puts the agent well past the target. Voice agents often cover this with a short filler phrase, such as "Let me check that for you," spoken while the tool runs.

For transport, browsers and mobile apps typically use WebRTC, which runs over UDP and is designed for realtime media. It tolerates packet loss gracefully instead of stalling on retransmissions the way TCP does. Server-to-server audio, such as streaming a phone call from a telephony provider, commonly uses WebSockets.

Turn Detection

The agent has to decide when the user has finished speaking. The simplest method is voice activity detection (VAD), which treats a period of silence as the end of a turn.

Silence alone is unreliable, though. People pause mid-sentence to think. A short silence threshold makes the agent cut users off, while a long one adds dead air to every turn.

Semantic turn detection improves on this by also looking at what was said. "My order number is..." followed by a pause is clearly incomplete, while "That's all, thanks" is clearly finished. A small model scores whether the utterance is complete and combines that with the silence duration. Systems often wait briefly after a complete-sounding sentence and longer after an incomplete one.

Some pipelines also start the LLM speculatively once a turn seems likely to be over, then discard the result if the user keeps talking. This trades extra model calls for lower latency.

Interruption Handling

Users interrupt, and the agent should stop speaking when they do. Handling this is called barge-in, and it takes several coordinated steps.

When the system detects user speech during playback, it stops playing audio immediately and flushes any buffered audio. It then cancels the in-flight LLM generation and TTS synthesis, which is the same upstream cancellation problem from text streaming but with a much tighter deadline.

The easiest step to overlook is truncating the conversation history. The LLM may have generated three sentences while the user heard only one before interrupting. If the history records all three, the model believes it said things the user never heard. The agent should record only what was actually played, which requires tracking how much audio reached the user.

Two practical issues complicate this. First, the agent's own voice coming out of a speakerphone can trigger VAD and make it interrupt itself, so echo cancellation is essential. Second, not every sound is an interruption. A cough or a brief "mm-hmm" shouldn't stop the agent mid-sentence, so many systems require a minimum speech duration or a few transcribed words before treating it as barge-in.

Streaming and realtime design is about controlling what the user experiences while the model is still working. Here are the main decisions and the default choices for each.

The common thread is that model output is slow, so the path from generation to the user needs to be designed deliberately.