Adding a feature that drafts customer replies can begin with a simple request to a language model. A support application sends the ticket and receives a string to display. That approach works until a request times out, a draft stops halfway through, or the application tries to read a refund amount from a sentence that changes between calls.

An API around a large language model (LLM) needs a contract for work whose duration, output, and cost can vary.

This chapter covers request modeling, conversation state, structured results, streaming, background execution, and the reliability decisions that make those patterns usable in production.

A language model generates output from supplied context. That context might include instructions, conversation messages, documents, or images. A token is a unit the model uses to represent input or output; it is not necessarily a word, and tokenization depends on the model.

Unlike a database lookup, repeating a generation request can produce different wording or different conclusions. The API can still provide predictable validation, authorization, lifecycle states, and result shapes. Those are responsibilities of the service, even when the generated content varies.

Separate three questions in the contract:

A successful HTTP exchange does not establish factual correctness. A well-formed draft does not authorize sending it to a customer.

There are also two different API products to consider. A general inference API exposes model capabilities to developers who need direct control. A task-specific API exposes a business operation, such as drafting a support reply, and owns the prompt and model selection internally. Neither is universally better. Choose based on what the caller needs to control and what your service can promise to maintain.

For the examples here, a support service owns a task-specific API. All paths, JSON fields, event names, model labels, and error codes shown below are illustrative application conventions, not a standard LLM protocol. Requests use HTTPS; credentials are placeholders. The authenticated identity determines tenant access.

The support API creates a generation resource. Its support_reply_v3 profile identifies a documented combination of task instructions, output rules, and permitted model capabilities. Clients choose the task contract without needing to supply the service's internal prompt.

Here is a synchronous request for a short draft:

The server validates the profile, authorizes access to the ticket, and loads an input snapshot before calling the model. The free-form instruction can guide wording, but cannot override access rules or supply trusted delivery facts.

A completed creation returns:

The model label and token counts are illustrative. resolved_model identifies the execution version this service recorded. input_revision identifies the ticket snapshot the service used for generation; it helps explain why a draft may differ from the ticket's current state.

The output is an array of typed blocks so clients do not have to assume every result is a string. A broader inference API might also support images, audio, or tool requests. Introduce those variants only when the product supports them, and document how clients handle unsupported types.

max_output_tokens bounds output using the service’s documented rules for counting model tokens. It does not specify an exact length, a word count, or a maximum total bill. Avoid exposing generation controls merely because an upstream provider has them. For example, a randomness control such as temperature needs model-specific semantics; setting it to zero is not a sufficient API guarantee of identical results.

The service must provide the capabilities it advertises for each profile. If a requested output format is unsupported, reject it before spending model capacity. Silently replacing structured output with prose would break the caller's expectations.

A model does not automatically know what a user said in a previous HTTP request. Some component must supply the relevant history or refer to state a service retains.

With client-managed history, each request includes the messages needed for the next generation. This gives the caller explicit control over the context, but increases payload size and makes the caller responsible for preserving order and staying within limits. A general inference API may use message roles such as user and assistant; exact roles and instruction precedence depend on its contract.

With server-managed history, the client supplies a conversation identifier and a new message. The server loads authorized history and constructs the model input. Requests become smaller, but retention, deletion, concurrent updates, and tenant isolation become service responsibilities. A conversation ID must be an authorized resource reference, not a bearer credential.

Our support API builds context from an authorized ticket snapshot rather than accepting arbitrary role-labeled history. This flow shows the trust boundary:

The server decides what data the caller may use before generation. Text inside a ticket or retrieved document remains untrusted content. Delimiters and message roles can help organize context, but do not establish authorization or eliminate prompt injection, where input content attempts to redirect model behavior.

A context window limits how much material the model can consider in one execution. The service must account for instructions, history, retrieved material, and output capacity using the selected model's rules. A small client payload can still lead to excessive model input after the server loads a long ticket.

Choose an explicit overflow policy. This API rejects requests whose assembled context exceeds its budget. Other services may select recent messages or summarize older material. If the service omits or summarizes history, expose that fact in execution metadata. Quietly dropping an early delivery constraint can change the meaning of a later draft.

For server-managed conversations, define how concurrent turns work. One useful rule requires an expected conversation revision and rejects a stale revision with a conflict. Another creates explicit branches. Without such a rule, two simultaneous messages can generate replies from different histories while appearing to belong to one ordered conversation.

Prose is appropriate when a person reads the result. A program needs a stronger contract when it must route a ticket, extract fields, or populate a form.

Asking the model to “return JSON” may improve formatting, but JSON syntax alone does not establish required fields or valid values. Structured output constrains generation to a supported schema. The service still needs to handle unsuccessful generation and validate the resulting values against business rules.

For this API, changing output_format to support_draft_v1 selects a server-owned schema. The format describes a draft object:

This is a schema fragment for the generated value, not the outer HTTP response. The service verifies that the chosen model supports the constraints it promises. Provider schema support varies, so a gateway should reject unsupported features instead of claiming arbitrary schema compatibility.

A successful structured result uses a typed block inside the generation's output array:

That object satisfies the schema, but needs_human_review: false is still a model-produced value. The application may require review regardless. Similarly, a generated refund amount can be a valid number and still violate refund policy.

Represent unsuccessful outcomes outside the success schema. In this API, a model refusal produces status: "refused", an empty output, and finish_reason: "refusal". Reaching the output limit produces status: "incomplete" and finish_reason: "output_limit". For structured formats, the service exposes no success value until it has a complete validated object. It must not insert fake required values to make a refusal or truncated result look valid.

If an implementation retries internally to repair malformed output, bound the repair attempts and include their cost in operational accounting. A valid response should not require an unpredictable number of hidden generations.

For an interactive draft editor, showing text as it arrives can reduce the user's wait for useful output. Streaming changes delivery; it does not guarantee earlier completion or make partial text safe to publish.

The support API lets a client request execution: "stream" with Accept: text/event-stream. It returns a resource location and an SSE body. Server-sent events (SSE) frame UTF-8 text events using fields such as event and data, with blank lines separating event blocks.

The following is an illustrative response header block followed by its decoded body. The example omits HTTP transfer framing:

These are application-defined events. A delta is a change to one output block, not necessarily one token, word, or network read. The client assembles deltas in order and treats generation.finished as the terminal application event. Its status determines success; merely receiving a terminal event does not.

The client uses a streaming HTTP client, such as browser fetch with incremental parsing, for this authenticated POST. The browser's native EventSource interface uses a GET connection and does not expose arbitrary request bodies or authorization headers. Using the SSE wire format does not require using that interface.

This exchange highlights what happens when generation fails after output begins:

The server cannot replace the HTTP status after sending response headers. It records the failure and, if the connection still works, emits a terminal event with status: "failed" and a machine-readable error. A disconnected stream without a terminal event leaves the outcome unknown to the client; the client retrieves the generation resource to reconcile it.

In this design, the service saves the final resource durably before sending the event that marks completion. Text shown during generation is provisional. Structured formats buffer the generated value until validation and do not expose partial JSON as a usable result.

The API does not promise stream replay. Retaining the final resource does not imply retaining every delta. Reconnection must not silently launch another generation or append a new attempt to the old text.

Bound stream duration and pending output buffers. Test flushing through the actual proxy path so small deltas reach the user promptly. If you must check content before showing it to users, buffer it for that check or return a complete result. You cannot take back text that users have already seen in a stream.

The animation switches from support drafts to an orders example. It shows an application calling an orders API for a model, then streaming the model’s answer.

Streaming keeps a connection open. Background execution allows the work to outlive that connection. A lengthy task that combines information from documents or a queue of support drafts often needs the latter even if the application also offers live progress.

This API accepts execution: "background" and returns:

The client polls GET /v1/generations/gen_844 using its normal authorization. A 200 OK means the client retrieved the status resource; the body might still report queued or running. This service retains generation resources for 24 hours after termination and authorizes every read.

The same lifecycle applies across delivery modes. This diagram uses the support service's chosen terminal states:

incomplete means generation produced only a partial result, such as when it reached an output limit. failed means an operational failure prevented an acceptable result. These distinctions tell clients whether to offer a shorter request, display a refusal, or investigate a service failure.

The service also offers POST /v1/generations/{id}/cancel. Cancellation is a request to stop remaining work, not proof that the model generated no tokens or the provider charged for none. Completion may win the race; the operation then remains completed. Repeated cancellation requests return the current state without restarting work.

For this API, disconnecting a client does not cancel an accepted generation in any mode. Work continues within its server deadline unless the client explicitly cancels it. This policy supports recovery after a network loss, but can consume capacity after a user closes the page. A service that chooses disconnect cancellation should state that policy just as clearly.

For independent bulk work, a batch API can accept many generation requests and return per-item outcomes. Correlate results by caller-supplied item identifiers rather than response order, and retry failed items individually. Batch processing, background execution, and streaming solve different needs and can coexist.

Reject invalid requests before model execution where possible. If context assembly exceeds this service's permitted input budget, it returns:

This API chooses 422 for a syntactically valid request that the service cannot process under its input rules. An oversized HTTP request body is a separate condition that can warrant 413 Content Too Large. Token budgets and byte limits measure different things.

An authenticated caller lacking the generation permission receives:

This rejection happens before loading ticket content. The API separately uses 404 Not Found for ticket or generation identifiers outside the caller's visible resources to avoid disclosing their existence. A model refusal is a generation outcome, not an authorization failure.

Retry Identity

A network timeout does not prove the provider did no work. Blindly repeating a POST can create two drafts and incur two inference charges.

The example service defines Idempotency-Key as an application contract scoped to tenant and operation. It atomically records the key and normalized request before dispatching work, retains that mapping for 24 hours, and returns the existing generation for a repeated matching request. Reusing the key with a different request returns 409 Conflict. A replay returns the current JSON resource rather than reopening a delta stream.

The original ticket snapshot stays attached to the generation even if the ticket changes between retries. A deliberately new draft uses a new key. Retrying after the retention window may create new work, so clients must not assume indefinite deduplication.

This prevents duplicate application resources within the stated window. It does not guarantee exactly one provider execution. If the provider accepted work but the adapter never received its response, the adapter needs provider-supported reconciliation or must preserve the uncertain outcome. It cannot safely claim that redispatching is free of duplicate cost.

Capacity and Usage

Request-count limits alone do not describe LLM load. One request may ask for a short sentence; another may carry a long document and consume substantial output capacity. Bound input size, output allowance, concurrent generations, queue time, and total execution time in addition to requests per interval.

Before acceptance, reserve capacity against the caller's budget. When execution ends, adjust the reserved budget to match measured usage. Distinguish estimates, provisional stream usage, and final usage; if upstream accounting is unavailable, report that uncertainty instead of recording zero. A cancellation or failure can still have usage.

Retries and model fallbacks must share a bounded overall deadline and spending allowance. Switching providers after already streaming text can also create contradictory output. Finish the current attempt as failed or expose an explicit replacement attempt rather than silently mixing their text.

A stable JSON schema does not guarantee stable behavior. Changing the model, instructions, context selection, or output schema can alter refusal frequency, factual accuracy, verbosity, latency, and cost.

Record those execution versions separately. The example response exposes a profile and resolved model; internal metadata also records instruction and schema versions. A mutable model alias is convenient for operations, but clients needing reproducibility need a documented change policy. Even a pinned version should not imply byte-identical output unless the service can support that guarantee.

Before changing the support profile, evaluate representative tickets for unsupported delivery promises, missing-information handling, structured-output validity, and review-routing quality. Include long tickets and adversarial content. A deployment that keeps returning valid JSON while inventing refund eligibility has degraded the product.

Measure time to first useful output separately from total generation time. Track terminal outcomes, token usage, queue delay, and provider attempts. Counting initial HTTP success alone misses generations that later fail or stop incomplete.

Keep diagnostic identifiers and outcome metadata without logging raw tickets or drafts by default. State what the application retains, what it sends to a provider, and how deletion works. Application retention and provider retention are separate policies you must check for the chosen deployment.

Caching also needs an explicit contract. Reusing a previous generated answer is different from a provider reusing computation for a repeated prompt prefix. An application result cache must account for tenant access, input revisions, profile and model versions, output format, and any generation settings that affect the result. Similar wording is not enough to establish that two tickets may safely share an answer.

A provider adapter can normalize common fields, but it should not hide unsupported capabilities, different token accounting, or different completion semantics. Keep the public promise narrow enough that every supported backend can honor it.

LLM APIs need predictable contracts around variable generation. Model requests and retained results explicitly, choose who manages context, and distinguish HTTP success from complete output and business correctness.

Use structured output for machine-readable results, streaming for early display, and background resources for work that must survive disconnection. Define terminal outcomes, retry identity, cancellation, and usage accounting so clients can recover without guessing. Treat model and prompt changes as behavioral changes that require evaluation, even when the response schema stays the same.