An LLM feature can start as just a few lines of code: a service stores an API key, calls a model provider, and returns the response. But as more features are added, things get complicated. Multiple services start calling different providers, each with its own retry logic, API keys, and usage tracking. A provider outage can affect several features, and tracking which team is responsible for a sudden increase in costs becomes difficult.

An LLM gateway solves these problems by providing a shared layer between applications and model providers. Every model request passes through the gateway, making it a central place to route requests, enforce rate limits, handle failures, cache responses, and track costs.

In this chapter, we'll walk through how an LLM gateway works and explore its key responsibilities: model routing, fallbacks, token-based rate limiting, caching, and cost tracking.

Without a gateway, every service calls model providers directly and handles authentication, retries, rate limits, and usage tracking independently. This leads to duplicated logic and inconsistent behavior across services.

An LLM gateway centralizes these responsibilities:

Applications call a single internal API, often compatible with a popular provider's API format so existing SDKs can work with minimal changes. The gateway handles everything between the application and the model provider.

Since every model request passes through the gateway, it must be fast and highly available. A typical deployment uses stateless instances behind a load balancer, with shared stores like Redis for rate-limit counters and cached responses.

The goal is to keep gateway overhead to a few milliseconds under normal conditions, which is small compared to model inference latency.

Tools like LiteLLM, Portkey, and Envoy AI Gateway provide many of these capabilities out of the box. But the underlying design decisions remain the same whether you build a gateway or use an existing solution.

How a Request Flows Through the Gateway

The order of operations matters. Authentication and quota checks happen early to reject invalid requests before doing expensive work. Cached requests should also avoid unnecessary provider calls.

First, the gateway authenticates the caller and tags the request with its team, feature, and tenant. It checks applicable quotas, selects a model, and looks for a cached response.

On a cache miss, it reserves an estimated token budget and calls the provider. Once the response completes, it reconciles the reservation against actual usage and records the cost. Usage events can be published asynchronously to keep observability work off the critical response path.

Each step introduces its own design decisions. Let's start with model routing, which determines where requests go and has a direct impact on response quality, latency, and cost.

Not every request needs the same model. Classifying a support ticket is much simpler than debugging a complex integration, and a large reasoning model can cost an order of magnitude or more per token than a small model.

Model routing helps reduce costs by sending each request to a model capable of handling it, without sacrificing quality where it matters. There are three common approaches.

Routing by Task Type

The simplest approach is to map each task type to a specific model. For example, task=intent_classification goes to a small model, task=answer_generation to a mid-sized model, and task=code_review to a reasoning model.

These mappings live in the gateway's configuration, so changing the model for a task doesn't require updating every application that uses it.

This works well for features with predictable requirements. The limitation is that every request within a task uses the same model, even when difficulty varies. A simple support question and a complex troubleshooting request might both go to the same expensive model.

Model Cascades

A model cascade handles this variation by trying a cheaper model first and escalating to a more capable model only when necessary.

The main challenge is deciding whether the first answer is good enough.

You can use JSON schema validation, token log probabilities, a small grader model, or task-specific tests. Schema validation is fast but only catches structural errors. A grader can assess answer quality more broadly, but adds latency and cost, and its judgments aren't always reliable.

Whether a cascade saves money depends largely on how often requests escalate.

Suppose a small model costs $0.001 per request, a grader costs $0.0005, and a large model costs $0.015. If 30% of requests escalate, the average cost is:

That's a 60% cost reduction.

But the savings shrink as more requests escalate. At a 90% escalation rate, the average cost reaches $0.015, making the cascade just as expensive as calling the large model directly. Escalated requests also take longer because they require two model calls and a quality check.

Cascades are also difficult to use with streaming. Once you've streamed part of an answer to the user, you can't easily discard it and start over with another model. This makes cascades better suited to background jobs, structured outputs, and short responses than long, streamed conversations.

Classifier-Based Routing

Instead of generating an answer with the small model first, classifier-based routing predicts which model should handle the request.

A lightweight classifier estimates whether the smaller model can answer the request well. Requests above a confidence threshold go to the smaller model, while more difficult ones go directly to the larger model.

The classifier can be trained using graded responses from both models on representative requests or human preference data. Research such as FrugalGPT and RouteLLM has shown that this approach can significantly reduce costs while maintaining quality close to that of a stronger model.

The confidence threshold controls the trade-off between cost and quality. A higher threshold sends more requests to the expensive model, while a lower threshold favors cost savings at the risk of weaker answers.

The right threshold should come from evaluating both cost and answer quality on your own workloads.

Classifiers also need ongoing maintenance. When models change or incoming requests become more complex, routing decisions can become less reliable. Monitor quality by route and periodically update or retrain the classifier.

Choosing a Routing Strategy

In practice, these approaches often work together. You might use task-based routing across features, then add a classifier or cascade for tasks where request difficulty varies significantly.

But routing assumes the selected model is available and working. Next, we'll look at how the gateway handles provider failures through retries and fallbacks.

LLM providers can experience timeouts, capacity shortages, and rate limits, especially during traffic spikes. Without proper fallback handling, these failures become errors for users. A well-designed gateway can recover from many of them by retrying or switching to another deployment or model.

Classify the Error First

Not every failure should trigger a retry. If a request is invalid, sending it to another provider will likely produce the same error.

For retryable errors, use exponential backoff with jitter to prevent clients from retrying simultaneously. Also enforce a retry budget, such as limiting additional retry attempts to 10% of normal request volume.

Without a retry budget, an overloaded provider can receive several times its normal request volume, making the problem worse. Retries should also respect the request's overall timeout so users aren't left waiting through multiple attempts.

The Fallback Chain

If retries don't resolve the problem, the gateway moves through a predefined sequence of alternatives:

The safest fallback is another deployment of the same model, since it minimizes changes in response quality and behavior. Some models are available through multiple cloud platforms, such as AWS Bedrock, Google Vertex AI, and Azure, providing alternative deployments with separate capacity.

Switching to a different model is more complicated. Prompts optimized for one model may not work as well with another, and providers can differ in tool calling, structured outputs, and other capabilities.

Every fallback model should pass the same evaluation suite as the primary model. The gateway may also need model-specific prompt templates or request transformations. Otherwise, a provider outage could lower response quality without raising any error.

Data residency requirements also constrain fallback options. If a customer's data must remain in the EU, switching to a US deployment isn't acceptable. Fallback chains should therefore respect each tenant's data residency and compliance requirements.

Circuit Breakers

Retries and fallbacks help with temporary failures, but repeatedly calling a provider that's already down wastes time.

A circuit breaker monitors failures for each provider and deployment. When the failure rate exceeds a configured threshold, the circuit opens and requests go directly to the next available option.

After a cooldown period, the gateway allows a small number of test requests through. If they succeed, the circuit closes and normal traffic resumes. If failures continue, it stays open.

This prevents every request from waiting for a known failing provider before falling back.

Fallbacks and Streaming

Streaming makes fallback handling more difficult.

Before the first token is sent to the client, the gateway can switch models without affecting the user. But once streaming begins, switching models mid-response could produce an inconsistent answer.

If a stream fails partway through, the gateway has two main options: return the partial response with an option to retry, or restart the request using a fallback model and instruct the client to discard the previous output.

In either case, the gateway shouldn't combine output from two different models without signaling it to the client.

Fallbacks help manage failures and capacity limits on the provider side. Next, we'll look at rate limiting and quotas, which protect shared capacity and prevent a single tenant or feature from consuming too many resources.

Traditional rate limiting counts requests, but LLM requests vary significantly in size. A classification request might use 50 tokens, while summarizing a large document could require 100,000 tokens. Both count as one request, even though their costs and resource requirements are very different.

That's why LLM gateways need token-based rate limiting.

Providers typically enforce tokens-per-minute (TPM) and requests-per-minute (RPM) limits, sometimes with separate limits for input and output tokens. If the gateway only counts requests, a single tenant processing large documents could exhaust the organization's token capacity and cause 429 errors for everyone else.

Reserving Tokens and Settling Usage

The challenge is that you know the input size before making a request, but not how many output tokens the model will generate.

A common solution is to reserve tokens before the request and reconcile usage afterward.

For example, suppose a tenant has a limit of 200,000 tokens per minute. A request contains 6,000 input tokens and sets max_tokens to 1,000.

The gateway initially reserves 7,000 tokens. If the model generates only 300 output tokens, actual usage is 6,300 tokens, so the gateway refunds the remaining 700.

These counters typically live in a shared store like Redis, with atomic operations to prevent concurrent requests from exceeding the same quota.

Quotas at Several Levels

A gateway usually enforces limits at multiple levels. Each request must satisfy every applicable limit before reaching the provider.

Rate limits and budgets solve different problems. Rate limits protect capacity over short periods, such as seconds or minutes. Budgets control spending over longer periods, typically days or months. For example, a feature might trigger an alert at 80% of its monthly budget and stop accepting requests at 100%.

When provider capacity is limited, the gateway should also prioritize important traffic. A common approach is to reserve capacity for interactive requests and process background jobs through a queue using the remaining capacity.

For workloads that don't need immediate results, provider batch APIs can offer lower costs in exchange for longer processing times.

Rate limits control how much traffic reaches a model provider. Caching goes a step further by avoiding unnecessary model calls altogether.

LLM gateways typically use three types of caching. Two reuse previously generated responses, while the third reuses computation performed by the model provider.

Each reduces costs in a different way, but they also differ in cache hit rates, latency savings, and the risk of returning incorrect or outdated answers.

Exact-Match Response Caching

The simplest approach is to cache the complete response for a given request.

The gateway generates a cache key by hashing everything that affects the response, including the model and its version, messages, system prompt version, tool definitions, and sampling parameters such as temperature.

When an identical request arrives, the gateway returns the cached response without calling the model again.

This works well for repetitive workloads, but its effectiveness depends on the application. In conversational systems, each new message changes the conversation history, so exact matches are rare. For structured tasks, such as extracting information from the same document or classifying repeated product titles, cache hits are much more common.

A few precautions are important:

Exact-match caching is relatively straightforward because requests with the same inputs can reuse the same answer, provided that the answer is still valid.

Semantic Caching

Semantic caching goes further by reusing responses for questions with similar meanings, even when the wording differs.

The gateway converts each query into an embedding and searches a vector index of previously cached queries. If the closest match exceeds a similarity threshold, such as 0.95 cosine similarity, the gateway returns the cached response instead of calling the model.

For example, these questions could share the same answer:

The challenge is that semantic similarity doesn't guarantee that two questions have the same answer.

Consider these examples:

The last two examples show that even identical questions can require different answers depending on user context or when they're asked.

To reduce these risks:

Lowering the similarity threshold generally increases cache hits, but it also increases the risk of returning incorrect answers.

Semantic caching works best for high-volume applications with repetitive, well-defined questions. Applying it indiscriminately to every request can introduce subtle errors that are difficult to detect.

Provider Prompt Caching

Unlike response caching, provider prompt caching doesn't skip model generation. Instead, it reuses computation from previously processed input tokens.

When a model processes a prompt, it computes intermediate attention states known as the key-value (KV) cache. If another request begins with the same tokens, the provider can reuse those cached states instead of processing the entire prefix again.

This reduces input processing costs and often improves time to first token.

At the time of writing, major providers, including OpenAI, Anthropic, and Google, support prompt caching, though the controls differ. Some cache eligible prompts automatically, some let you mark explicit cache breakpoints, and several offer both.

Cached input tokens can be significantly cheaper than regular input tokens. Some providers charge around 10% of the normal input price for cache reads, while charging a premium to create new cache entries. Cache lifetime also varies by provider and model, ranging from minutes to longer configurable periods.

The key requirement is that cached content must match an exact prompt prefix. If a token changes near the beginning, the content after that point may no longer be reusable.

This makes prompt structure important:

Place content that stays the same across requests first, followed by session-specific context and then content that changes with each turn.

This works particularly well for conversations because new messages are typically appended to the history, allowing later requests to reuse much of the previous prompt.

Small changes can significantly reduce cache efficiency. Adding a timestamp to system instructions, changing the order of tool definitions, or placing user-specific information near the beginning of a shared prompt can prevent prefix reuse.

To catch these problems, monitor the cached-token ratio, calculated as cached input tokens divided by total input tokens. A sudden drop after a deployment often indicates that something changed in the prompt structure or routing behavior.

Example: Estimating Prompt Caching Savings

Suppose a customer support assistant processes 1 million requests per day. Each request contains:

Assume 90% of requests reuse the shared prefix, cached reads cost 10% of the normal input price, and cache writes cost 125%.

That's about a 65% reduction in input token costs, without changing the generated answers. Output token costs remain unchanged in this example. These are illustrative assumptions, and actual cache pricing and hit rates depend on the provider, model, and cache configuration.

Routing decisions also affect caching efficiency. Provider-side caches are generally tied to a particular model and deployment infrastructure. If the gateway distributes related requests across multiple providers or deployments, each may need to build its own cache.

Sticky routing can improve reuse by keeping requests that share a prefix on the same deployment. For self-hosted models, prefix-aware load balancing applies the same principle at the inference replica level. Managed providers may handle cache-aware routing internally.

Choosing a Caching Strategy

When prompts share long, stable prefixes, provider prompt caching is a good place to start. It can deliver substantial savings without the answer-reuse risks of response caching.

Add exact-match caching for repetitive workloads, and consider semantic caching only when you can reliably measure and control incorrect cache hits.

Caching reduces costs, but it also makes request-level cost accounting more complicated. Next, we'll look at how the gateway tracks usage and attributes costs across teams, tenants, and features.

A provider invoice shows the total cost for an account, but that isn't enough to understand where the money is going. Teams often want spending broken down by feature or by user, and a B2B company may need cost per customer to price its plans.

Since every model request passes through the gateway, it's the natural place to track and attribute these costs.

Tag Every Request

Each request should include metadata such as team, feature, tenant, user, environment, and prompt version.

Internal API keys can identify the calling service automatically, while applications supply additional context such as feature and tenant_id. The gateway should validate required fields and reject requests that don't include them. Otherwise, a significant portion of spending can end up attributed to "unknown".

Calculate Costs From Actual Usage

Cost calculations should use the token counts reported by the provider rather than the gateway's initial estimates.

The gateway maintains a pricing table for each provider and model version, accounting for input tokens, output tokens, cached input, and cache writes where applicable. Reasoning tokens may also contribute to billed output usage, even though they aren't visible to the user.

The gateway publishes usage events to a queue, allowing cost processing to happen asynchronously without delaying responses. A cost processor applies the correct prices and stores the results in a warehouse, where teams can analyze spending by feature, tenant, model, or any other tag.

This data also supports budget alerts and spending controls, although hard budget enforcement needs more immediate counters rather than waiting for warehouse updates.

Usage events should have unique IDs to prevent duplicate charges during event retries. Periodically reconciling recorded costs against provider invoices helps catch pricing errors, missing usage, and requests that bypassed the gateway.

Edge Cases

Some workloads need additional accounting rules:

With accurate attribution, a product manager can see that a feature costs $0.04 per conversation, compare that against revenue, and decide whether further investment in routing, caching, or model optimization is worthwhile.

An LLM gateway centralizes model access, routing, reliability, rate limiting, caching, and cost tracking. Instead of implementing these capabilities separately in every service, teams can manage them through a shared layer.

Here are the key design decisions:

In a system design interview, start with the gateway capabilities your application actually needs, such as provider failover, per-tenant quotas, or usage tracking. Add more advanced routing and caching strategies when traffic patterns, quality requirements, and cost measurements justify the added complexity.