When a company starts building with LLMs, each team often integrates with model providers on its own. One team might call OpenAI with a key stored in an environment variable, another might use Anthropic models through AWS Bedrock, and a third might serve an open-weight model on its own GPUs. Each team writes its own retry logic and handles rate limits in its own way.

As usage grows, this causes problems across the company. A provider outage can take down every product that has no fallback. Finance may see the monthly bill go up without being able to tell which team is responsible. Prompts that contain customer names and phone numbers can end up in debug logs that many people can read. A gateway solves these problems by putting one shared layer between all applications and the model providers.

In this chapter, we'll design an LLM gateway for a large company. We'll start with the requirements and high-level architecture, then go deep on token-based rate limiting, provider failover, caching and its risks, and keeping gateway latency low.

Let's start by understanding the problem.

An LLM gateway is a reverse proxy between internal services and model providers. Applications send requests in one standard format, and the gateway decides which provider and model serves each request, enforces limits, records usage, and returns the response.

Without a gateway, every application holds its own provider credentials and integration code, and policies like quotas, logging rules, and fallbacks exist only if each team builds them. With a gateway, applications only know about the gateway. Provider keys live in one place, and a policy change applies to every caller at once.

Many gateways expose an OpenAI-compatible API, since many SDKs and frameworks already support that format. A call through the gateway looks like an ordinary chat completion, except that the application asks for a model alias instead of a specific provider model:

The bearer token is a gateway-issued virtual key, not a provider key. It identifies the team and application and carries the quotas and data policies attached to them. The alias chat-large maps to a list of real deployments controlled by the platform team, so traffic can move between providers without any team changing code. The extra headers let the gateway attribute cost to a feature and an end customer.

Because the gateway sits on the critical path of every AI feature in the company, its availability becomes a ceiling for everyone else's, and every millisecond it adds shows up in every product.

A gateway can be a thin proxy or a full platform with quotas, fallbacks, and logging rules. That's why it's worth clarifying the scope before getting into the design.

Here is one way the discussion could unfold:

This conversation sets several constraints for the design. Let's formalize these into functional and non-functional requirements.

Functional Requirements

Non-Functional Requirements

Scale Estimates

Let's assume 20 million chat completion calls per day from interactive products, internal tools, and batch jobs.

These numbers are small for a web service, but LLM requests are long. If a streaming request stays open for five seconds on average, peak traffic means about 3,500 concurrent connections. The gateway spends almost all of that time waiting on providers, so the work is I/O-bound. Redundancy across zones and regions decides the instance count more than CPU does.

Cost is the bigger number. Assume 2,000 input tokens and 300 output tokens per call, at a blended illustrative price of $2 per million input tokens and $8 per million output tokens:

At this level, small percentages matter. Saving 2% of tokens through caching is worth about $77,000 per month, and attribution that's off by 5% assigns $190,000 a month to the wrong teams.

Logging adds up too. A metadata record per call is about 1 KB, or 20 GB per day. At roughly 4 characters per token, a full 2,300-token exchange is about 9 KB of text, so logging every payload adds around 200 GB per day, or about 6 TB of sensitive text over 30 days of retention. That's a strong reason to make full payload logging opt-in.

Most gateway metrics look like those of any proxy. A few are specific to LLM traffic.

Some of these pull against each other. Every check the gateway runs, such as quota lookups, cache lookups, and PII scanning, adds overhead. The design has to keep the hot path short and move everything else off it.

The gateway has two halves. The data plane handles live requests and must be fast and highly available. The control plane stores configuration, such as virtual keys, routes, quotas, and logging policies, and pushes it to the data plane. Usage data flows out of the data plane asynchronously.

Requests flow left to right: the proxy layer checks quotas and the cache, then the routing engine picks a target and an adapter calls it. The dotted lines are off the request path. Usage events leave after the response, and configuration arrives ahead of time. Let's look at each component.

Proxy Layer

The proxy layer is a fleet of stateless servers that terminate client connections. For each request, it authenticates the virtual key, attaches the caller's policies, checks quotas, consults the cache, and streams the provider's response back.

All configuration lives in memory. Looking up a key in a database on every request would add latency and make the database a hard dependency, so the control plane pushes the full set of keys, routes, and policies to every instance. A few thousand keys and a few hundred routes fit easily in memory.

This also means the data plane keeps working when the control plane is down. Proxies keep serving with the last configuration they received. Teams can't create new keys during that window, but existing products keep running.

Routing Engine

The routing engine turns a model alias into a specific target: a provider, a region, and a model version. Each alias lists its targets in order of preference, and the engine picks one based on:

Cost-based routing should be owned by the team behind the alias, after they've evaluated both models on their own traffic. If the gateway swaps models for every team without telling them, product behavior can change in ways nobody tested.

Provider Adapters

Each provider has its own request format, streaming events, error codes, and way of reporting token usage. Adapters translate between these and the gateway's standard format.

This is more than renaming fields. Tool call formats differ, usage sometimes arrives only in the final stream event, and some providers report cached input tokens separately at a different price. The quota service and usage pipeline depend on accurate token counts, so the adapters have to get these details right.

Quota Service and Cache

The quota service tracks token consumption per scope (organization, team, application, end user) and per upstream deployment, and answers one question on the hot path: can this request be admitted now? The cache returns stored responses without calling a provider. Both have enough subtlety to get their own deep dives.

Usage Pipeline

Every request produces a usage event when it finishes:

The proxy buffers these events locally and ships them to a message queue such as Kafka. Consumers aggregate them for dashboards, compute cost from a versioned price table, and write results to a data warehouse for finance.

Cost is computed in the pipeline rather than in the proxy. Prices change, and cached, batch, and long-context tokens are often priced differently, so keeping pricing in one place makes it possible to recompute history when a price was wrong. A monthly job compares computed totals against provider invoices. A gap above about 1% often points to an adapter that miscounts tokens.

Logging with PII Controls

Payload logs help engineers debug bad answers and help evaluation teams build test sets. They are also the most sensitive data the gateway handles, so logging is a per-key policy rather than a global switch.

Redaction runs asynchronously in the logging pipeline, not on the request path. Pattern matching catches structured values like emails, phone numbers, and card numbers, and a named entity recognition model catches names and addresses. Each match is replaced with a placeholder like [EMAIL_1] before anything reaches long-term storage, and raw payloads wait in an encrypted buffer with a short time to live until then.

Redaction misses things, so the other controls matter just as much. Logs are encrypted, stored in the caller's region, readable only by the owning team and approved reviewers, deleted after the retention period, and every read is audited.

The Request Lifecycle

Here's how a streaming request moves through the data plane:

Authentication, policy lookup, and target selection happen in memory. The quota reservation is the only step that might need a network call before the provider request, and the latency deep dive shows how to avoid most of that cost. The usage event is sent after the response finishes, so it never delays the client.

A requests-per-minute limit protects very little with LLM traffic. A request with a short question and one with a 150,000-token document each count as one request, but the second uses hundreds of times more capacity. Providers enforce limits in tokens per minute for this reason, and the gateway needs to do the same.

The difficulty is that the gateway doesn't know a request's size when it arrives. It can count input tokens up front, but output tokens are only known after the model finishes.

Estimating Tokens Up Front and Reconciling Later

A common approach reserves an estimate at admission and corrects it on completion:

Input tokens come from the target model's tokenizer, or from a character-based estimate when the tokenizer isn't available. A rough count is acceptable because reconciliation fixes it.

The output estimate is the real trade-off. Reserving the full max_tokens prevents any overshoot, but many applications set max_tokens far above what they use, so their reservations block capacity that's never consumed. Reserving a typical output length admits more traffic but allows brief overshoots when responses run long. A practical middle ground is the smaller of max_tokens and the route's recent p90 output length:

If the counters live in Redis, try_reserve_all can be a single Lua script that checks every scope and increments them only if all have room. The rejection names the scope that ran out, so the caller knows whether one user hit a limit or the whole team did.

Tenant Limits vs. Upstream Capacity

The scope list mixes two kinds of limit that are worth keeping separate.

Tenant limits protect teams from each other. They are policy, set by the platform team and the teams themselves.

Upstream capacity is what the company has actually bought. If the contract with a provider allows 20 million tokens per minute on a model, sending more gets 429s for everyone. The gateway tracks each deployment's consumption as a shared pool.

Tenant limits usually add up to more than upstream capacity, since teams don't usually peak at the same time. When the pool gets tight, the gateway needs a rule for who slows down first. Tagging traffic by priority handles this well:

The tenant check comes first, so a team that used up its own limit gets a clear answer. The priority check only applies under shared pressure, letting a nightly batch job slow down instead of degrading a customer-facing chat.

Budgets and Long Streams

Dollar budgets accumulate over a month, so they're checked against reconciled spend from the usage pipeline rather than reservations on the hot path. The pipeline updates month-to-date spend every minute or so, alerts the owning team at a soft limit such as 80%, and has the control plane push a "budget exceeded" flag to the proxies at the hard limit. Being a minute behind lets a team overspend slightly, which is a fair price for keeping budget checks off the request path.

Streams create one more edge case. If a response runs far past its reservation, the tenant crosses its limit mid-stream. Cutting the stream would hand the user a broken answer, so the usual choice is to let it finish and charge the overage to the next window.

Hosted model APIs fail in several ways. They return 429s when the company's limit is reached, 5xx and overload errors during capacity crunches, and sometimes they stay up but slow down until TTFT goes from under a second to fifteen. Regional and full provider outages happen too.

The gateway's job is to hide these failures from the application when it can, and to return a fast, clear error when it can't.

Matching the Response to the Failure

The last row matters. A gateway that retries refused requests on other providers until one complies becomes a tool for getting around safety policies.

Ordering the Fallback Chain

The safest fallback is the same model hosted somewhere else. At the time of writing, OpenAI models are available through both OpenAI and Azure, and Anthropic models through Anthropic's API, AWS Bedrock, and Google Cloud Vertex AI. Switching hosts keeps the same model behavior. Switching to a different model family is riskier, because prompts tuned for one model can produce different output, different tool calls, or different JSON on another.

A route configuration reflects that ordering:

The first two targets split traffic across regions of the same deployment. The third is the same model on a different provider. The fourth is a different model family, used only by callers who have run their evaluation set against it and opted in. The data classes limit where data can go. A request tagged restricted never goes to an unapproved target, even during an outage, because failing the request is better than sending restricted data somewhere it isn't allowed.

Fallback targets also need a context window at least as large as the request. Falling back from a 200,000-token model to a 32,000-token model turns an outage into a stream of context-length errors.

Circuit Breakers

Without memory of recent failures, every request would try the broken primary, wait for the timeout, and only then fail over. Circuit breakers avoid this by tracking each target's health and skipping unhealthy ones.

Traffic flows normally while the breaker is closed. When the error rate or p95 TTFT over a sliding window crosses a threshold, the breaker opens and the routing engine skips that target. After a cooldown of around 30 seconds, it goes half-open and lets a small share of traffic through. Success closes it again, and failure reopens it.

Two details make breakers useful for LLM traffic. They track latency as well as errors, because a provider that returns HTTP 200 after twenty seconds is effectively down for an interactive product. And they're kept per provider, region, and model, because outages can hit one model in one region while everything else works. Each proxy instance can keep its own breaker state, since at hundreds of requests per second each instance sees enough traffic to detect a failure within seconds.

Avoiding Retry Storms

Failover can make an outage worse. If every request makes three attempts, a struggling provider receives triple the traffic at the worst moment, and the fallback provider gets flooded next. A few controls prevent this:

Failover and Streaming

Failover is invisible only before the first token reaches the client. While the gateway is still waiting for that token, it can try the next target, and the client just sees a slower response.

Once tokens have been streamed, switching providers would start a new answer from scratch in the middle of the old one. At that point, the gateway ends the stream with an error event, and the application decides whether to retry. This is one reason to keep the first-token timeout tight, since it marks the last moment when failover goes unnoticed.

At $3.8 million a month, every request answered without a model call saves money and time. There are three kinds of caching, with very different risks.

Exact-Match Caching

The simplest cache stores a response under a hash of everything that affects it: the resolved model, the full message list, tool definitions, and parameters like temperature and max_tokens. The key also includes the tenant, so two teams sending the same prompt never share an entry.

This cache is safe, but it rarely hits on interactive traffic, since conversation history, timestamps, and user names make requests unique. It works well for workloads that repeat: evaluation runs over a fixed test set, batch jobs that reprocess documents, and classification of a fixed set of inputs. It should be limited to temperature 0 requests or callers that explicitly enable it.

Semantic Caching

A semantic cache matches requests by meaning. It embeds the incoming request, searches a vector index of earlier requests, and returns the stored response when the nearest match is above a similarity threshold.

That raises the hit rate, and it also returns wrong answers. Embedding similarity measures how related two texts are, which is different from whether they ask the same question:

A higher threshold cuts errors but also removes most hits, and a threshold tuned for one application may not suit another. A reliable way to set it is to sample cache hits, compare each cached answer with a fresh one using an LLM judge or human review, and track the error rate per route.

For a company-wide gateway, this means semantic caching should be off by default. A team can enable it for a route when the content isn't personalized, the cache is scoped to that route, the key includes a hash of the system prompt and retrieved context, and the measured error rate is acceptable. Answers to common questions in a public help center are a reasonable fit. A support bot that looks up a customer's account is not.

The lookup also has a cost. An embedding call plus a vector search adds roughly 10 to 30 ms to every request on the route, while the savings come only from the hits.

Provider Prompt Caching

Providers offer a different kind of caching. When requests share a long identical prefix, such as a system prompt, tool definitions, or a large document, the provider reuses the computation for that prefix. Cached input tokens are billed at a large discount, and TTFT drops because the prefix isn't processed again. Some providers do this automatically for long prompts, while others require the caller to mark where the reusable prefix ends.

This caching never returns a wrong answer, since the model still generates a fresh response. The gateway can't create these cache hits, but it can help applications get more of them:

Comparison

Provider prompt caching often gives the largest saving with the least risk, so the gateway should support it everywhere. Exact-match caching is a cheap opt-in for batch and evaluation traffic. Semantic caching belongs only on a few routes where its error rate has been measured.

Staying under 10 ms of overhead at p99 is achievable, but only if the hot path avoids anything slow.

Sources of Gateway Latency

Here's a rough comparison between a gateway that does everything synchronously and one built for low overhead:

The expensive items in the synchronous column are network calls and blocking work the request doesn't need to wait for. The optimized design moves them off the path rather than making them faster.

Reducing Gateway Latency

Local quota leases. Instead of calling a central counter on every request, each proxy instance leases a slice of each tenant's limit, such as 5% of the team's tokens per minute, and admits requests against it locally. When a lease runs low, the instance fetches more in the background. Leases expire on their own, so a crashed instance doesn't hold capacity for long. The cost is precision: a tenant can briefly exceed its limit by roughly the size of its outstanding leases. For small tenants with low limits, the gateway can skip leases and check the central counter directly.

Pooled upstream connections. A new TLS connection to a provider in another region can cost 50 to 150 ms. Proxies keep warm, long-lived connections to every provider endpoint so requests reuse them.

Unbuffered streaming. The proxy forwards each chunk to the client as soon as it arrives. It parses the stream incrementally, just enough to read the final usage event, but never waits for the full response. Buffering to inspect the response would erase the benefit of streaming.

Side work off the path. Usage events and logs go into an in-memory buffer and ship in batches. If the buffer fills, the proxy drops payload logs before it delays a request. Usage events drive billing, so they spill to local disk instead of being dropped.

Placement. The gateway should run in the same regions as its callers and the provider deployments it uses. A gateway on another continent can add more network latency than all of its own processing combined.

Failing Open or Closed

Keeping the hot path lean also means deciding what happens when a dependency fails:

The pattern is consistent. Availability of AI features takes priority over precise quotas, and data protection takes priority over debugging convenience.

A related choice is whether to run the gateway as a central service or as a sidecar beside each application. A sidecar saves a network hop of a millisecond or two, but it spreads provider credentials, quota state, and connection pools across thousands of processes. A central fleet in each region is usually the better choice, since the extra hop is tiny next to model latency and a central fleet is far easier to secure and operate.

Since every LLM call passes through it, the gateway becomes the best source of information about how the company uses models. A few signals deserve dashboards and alerts:

Aliases also make model upgrades safer. The platform team can send a small share of an alias's traffic to a new model version, compare quality and latency using the same logs and evaluation sets, and shift traffic gradually. Teams that pin a specific version keep it until their own evaluations pass.

At ten times the traffic, the proxies scale horizontally without much trouble. Upstream capacity is likely to run out first, which means negotiating higher limits, adding regions, or moving steady high-volume workloads to self-hosted models behind the same aliases. Payload logs grow next, and they're often sampled instead of captured in full.

An LLM gateway gives a company one place to control how it uses models: who can call them, how much they can spend, where data can go, and what happens when a provider fails.

The request path handles authentication, quota checks, routing, and forwarding, while logging, redaction, and usage metering run asynchronously. Because every AI product in the company depends on the gateway, it serves from the last known config when the control plane is down, fails open on quota with local leases, and buffers usage data locally if the queue is unavailable.