Unlike traditional infrastructure, LLM costs are heavily influenced by application design decisions. A longer system prompt, an extra retrieved document, a more detailed response, or another agent step adds a small cost to each request. Across millions of requests, these costs add up and can significantly affect a product's profitability.

In this chapter, we'll break down the main sources of cost in a GenAI system and explore practical ways to reduce them.

Before optimizing costs, you need to understand where the money is going. GenAI systems usually have six main cost sources:

For API-based applications, input, output, and reasoning tokens are usually the main variable inference costs, making them good starting points for optimization.

Consider a typical request in a RAG chat application:

At illustrative prices of $3 per million input tokens and $15 per million output tokens, this request costs $0.021.

The user sent just 100 tokens, but the system processed 5,000 input tokens and generated another 400. Input accounts for more than 70% of the total cost, with 2,000 tokens coming from system instructions and tool definitions that may be repeated across requests.

Additional Model Calls per Request

The main model call is often just one part of the cost. A single user request might also involve query rewriting, embedding generation, reranking, safety checks, and retries when a response fails validation.

Background processes add more costs, including LLM-as-judge evaluations, conversation summarization, and document ingestion pipelines.

Agents can be particularly expensive because every step in the tool-calling loop may resend the accumulated context, including instructions, tool definitions, and previous tool results.

For example, an agent that runs for 20 steps, with its context gradually growing to 30,000 tokens, might process roughly 300,000 input tokens in total. At $3 per million tokens, that's $0.90 in input costs alone.

If the agent gets stuck in a loop and executes 200 steps instead, the cost can grow dramatically, especially if the context continues expanding.

Measure Before You Optimize

The first step is tracking costs accurately. Every model call should record the feature, model, user or tenant, input tokens, cached tokens, output tokens, and estimated cost.

The LLM gateway is a natural place to collect this data because model requests already pass through it.

With proper tracking, you can answer questions that help identify optimization opportunities:

However, reducing cost per token doesn't necessarily improve business outcomes.

The metric that matters most is cost per successful outcome, such as cost per resolved support ticket or accepted code suggestion.

A cheaper model that resolves fewer support tickets might actually cost more per successful resolution because it requires additional attempts or human intervention.

Once you understand where costs come from and how they affect outcomes, you can focus on optimizations that deliver the most value.

There are several ways to reduce GenAI costs, each targeting a different part of the bill. Some, like prompt caching, are relatively easy to implement. Others, like model routing, require careful evaluation to ensure cost savings don't come at the expense of response quality.

Smaller or Routed Models

One of the biggest opportunities for cost reduction is choosing the right model for each task. Smaller models can cost an order of magnitude or more less per token than the largest frontier models, and many requests don't need the most capable model available.

There are two common approaches:

A typical routing architecture looks like this:

The router can use simple rules, a trained classifier, or a small LLM. Straightforward requests go to the cheaper model, while more complex ones go directly to the larger model.

If the smaller model fails a validation check, such as schema validation or a task-specific confidence threshold, the request can be escalated to the larger model.

This fallback helps maintain quality, but it also means paying for two model calls. If too many requests are escalated, the additional cost can offset the savings from routing.

How much routing saves depends on the share of traffic a smaller model can handle reliably and on how often requests get escalated. When most traffic is simple, the reduction in model spending can be substantial.

The biggest risk is a gradual decline in response quality that goes unnoticed. Before introducing routing, build an evaluation dataset from real requests and compare model performance across different task categories. Continue monitoring quality and user feedback for each route after deployment.

Prompt Caching and Shorter Prompts

Most LLM requests contain repeated content, such as system instructions, tool definitions, few-shot examples, and shared documents. Without caching, these tokens are processed and billed at the regular input rate on every request.

Prompt caching allows the model provider to reuse previously computed internal state for a matching prompt prefix. Cached tokens are typically billed at a discount, often 50 to 90% below the regular input price, depending on the provider and caching mechanism.

Caching can also reduce time to first token because the model doesn't need to process the cached prefix again.

The key requirement is that the prefix must match. If you place changing information, such as a timestamp or user-specific data, near the beginning of the prompt, you reduce the portion that can be reused.

That's why prompt structure matters:

Place static content first, followed by conversation history and request-specific context.

This gives repeated requests the best chance of sharing a cached prefix. In multi-turn conversations, previously processed messages may also remain cacheable if the prompt is constructed consistently and new messages are appended without changing earlier content.

Cache lifetimes vary by provider. Some caches expire after a few minutes of inactivity, while others support longer retention. Requests arriving after expiration may need to process the full prompt again.

Caching reduces the cost of repeated tokens, but removing unnecessary tokens can save even more.

Common techniques for shortening prompts include:

One trade-off with conversation summarization is that it changes the prompt prefix. Frequent summarization can invalidate cached content, so it's often better to summarize at defined checkpoints rather than after every message.

Prompt caching savings depend on how much of each prompt repeats across requests. Cached input tokens are billed at a steep discount, so workloads with long shared prefixes can see large reductions in input cost, while prompts that change from the first token see little benefit. Shortening prompts saves additional costs in proportion to the tokens removed.

Response Caching

Prompt caching reduces the cost of processing repeated input, but the model still generates a new response.

Response caching goes further by skipping the model call entirely and returning a previously generated answer.

There are two common approaches:

For example, "How do I cancel my order?" and "How do I cancel my subscription?" may have similar embeddings even though they require completely different answers.

Response caching works particularly well for:

It's less effective for open-ended conversations, personalized recommendations, or requests that depend on frequently changing data.

Every cache hit avoids a model call, so when requests have similar costs, the savings roughly track the hit rate, minus cache overhead. For example, a 30% hit rate would cut model spending by about 30%. FAQ-heavy applications tend to reach higher hit rates, while open-ended chat applications may see very little benefit.

Batch APIs for Offline Work

Not every LLM task needs an immediate response.

Workloads like nightly report generation, evaluation runs, document classification, data labeling, and embedding backfills can often tolerate delays.

Many model providers offer batch APIs that process large collections of requests asynchronously at a discounted rate.

Instead of sending individual requests through the real-time API, you submit a batch of requests and retrieve the results when processing finishes.

The main optimization opportunity is identifying work that doesn't need to run synchronously. Moving those jobs to batch processing can reduce costs without changing the underlying model or output quality.

Limiting Output Length

Output tokens are often several times more expensive than input tokens. They also contribute directly to latency because autoregressive models generate output sequentially.

Longer responses aren't always better. Models may repeat information, overexplain simple questions, or include unnecessary introductions and summaries.

There are three ways to control output length:

Reasoning models need an additional control.

These models may generate internal reasoning tokens before producing the visible response. For simple tasks, a high reasoning budget can add significant cost without improving the result.

Set reasoning effort based on task complexity rather than using the same high setting for every request.

The savings depend on how verbose the current responses are. Concise output instructions help most when answers restate the question or add summaries nobody reads. Adjusting reasoning budgets can save more when requests use more reasoning tokens than they need.

Quantization and Higher Batch Sizes When Self-Hosting

Self-hosted models have a different cost structure.

Instead of paying a fixed price per token, you pay for GPU capacity. The effective cost per token depends on how efficiently you use that capacity.

A simplified calculation is:

This example assumes the same input workload and excludes other infrastructure costs. In practice, prompt processing, GPU utilization, and serving overhead also affect the total cost.

The main goal is to increase useful throughput per GPU while maintaining acceptable latency.

Two techniques do most of the work.

Quantization

Quantization reduces the precision used to store model weights, lowering memory requirements.

Moving from FP16 to FP8 roughly halves the memory required for weights, while INT4 reduces it to about a quarter, excluding quantization metadata.

For example, a 70B model that requires four GPUs under one FP16 deployment configuration might fit on two GPUs with FP8, depending on GPU memory and serving overhead.

Lower weight memory usage can also free up space for the KV cache, allowing more concurrent requests on the same hardware.

FP8 often preserves model quality well on supported hardware, while 4-bit quantization requires more careful evaluation because it can introduce larger accuracy losses on some tasks.

Larger Batch Sizes

Batching allows the GPU to process tokens for multiple requests together.

LLM decoding is often limited by memory bandwidth. Generating one token requires accessing model weights, which is expensive relative to the computation performed for a single request.

With batching, the GPU can reuse those weights across multiple requests in the same decoding step, improving overall throughput.

Serving engines like vLLM and SGLang support continuous batching, which dynamically adds and removes requests as they arrive and finish. This keeps the GPU better utilized without waiting for a fixed batch to complete.

However, larger batches also consume more GPU memory and can increase latency under heavy load. Beyond a certain point, additional concurrency may reduce the token generation speed experienced by individual users.

The goal is to find the batch configuration that maximizes throughput while meeting latency targets, particularly time to first token and time per output token.

Other useful optimizations include prefix caching in the serving engine, autoscaling during periods of low traffic, and using spot or preemptible instances for workloads that can tolerate interruptions.

The gains depend heavily on the model, hardware, and workload. On supported hardware, FP8 can raise throughput per GPU in memory-bound workloads, and continuous batching with tuned concurrency can deliver several times the throughput of a poorly utilized deployment.

A practical approach is to start with optimizations that have minimal impact on quality, such as prompt caching, batch processing, and removing unnecessary tokens. Then evaluate techniques like model routing, semantic caching, and quantization, where savings need to be balanced against accuracy and performance.

So far, we've focused on reducing average costs. But production systems also need protection against unexpected spending, whether it's a single expensive request, a user generating excessive traffic, or an agent stuck in a long-running loop.

Average cost per request might look reasonable on a dashboard, while a handful of expensive requests drive up the bill. A user automating thousands of requests through your free tier, an agent stuck in a retry loop, or a request generating far more tokens than expected can consume a significant amount of your budget in a short time.

To prevent this, you need cost limits at three levels.

Per Request

Every model call should have clear limits on how much work it can perform:

These limits keep individual requests predictable and prevent unusually expensive calls.

Per User or Tenant

User-level and tenant-level budgets protect against abuse and help keep pricing plans profitable.

Limits are typically defined in tokens, requests, or dollars per day or month, with different allowances for free and paid plans.

Reaching a limit doesn't always have to mean blocking the user. Applications can degrade gracefully by switching to a cheaper model, restricting expensive features like deep research, or reducing the allowed request rate.

This lets users continue using basic features while keeping costs under control.

Per Agent Run

Agents need additional limits because the number of model calls and tool executions isn't always known in advance.

Each agent run should have a budget covering:

When an agent reaches its budget, it should stop gracefully rather than failing without explanation.

For example, a coding agent can return the changes it has completed, save its progress, and ask the user whether to continue with a larger budget.

This matters most for long-running agents, where partial results may still be valuable.

Where to Enforce Limits

Request and user-level limits are best enforced centrally, usually at the LLM gateway. Agent-level step limits and timeouts belong in the agent orchestrator, which can share budget information with the gateway.

Before sending a model request, the gateway estimates its maximum cost based on input tokens, the output token limit, and the selected model's pricing.

It then checks the available budget, reserves the estimated amount, and sends the request. Once the response arrives, the gateway records the actual cost and releases any unused reservation.

Reserving the budget before execution is important when multiple requests run concurrently.

For example, suppose a user has $1 remaining in their budget and sends ten requests simultaneously, each costing up to $0.50. If the gateway only checks completed spending, all ten requests might pass the check and spend $5.

With atomic budget reservations, only two requests can proceed at that estimated cost.

Here's a simplified implementation for tracking an agent run's budget:

Before each model call, the agent reserves its estimated maximum cost. After the call, it settles the reservation using actual usage. Failed or interrupted calls also need reconciliation, since they may have incurred charges.

This example assumes a single-threaded agent. In production, reservations must be atomic when multiple requests or agent workers share a budget, and estimates should account for billable reasoning tokens and other usage charges.

When a limit is reached, the orchestrator catches BudgetExceeded, saves the agent's progress, and returns a useful status to the user.

Finally, add spending alerts. Monitor total spend, cost per request, cache hit rates, and unusual increases in token usage.

A sudden increase in cost per request might reveal a prompt change that broke caching, an unexpectedly long conversation history, or an agent repeatedly calling the same tool. Catching these issues early can prevent expensive surprises at the end of the month.

Now that we've covered how to reduce costs and keep them within limits, let's apply these techniques to a practical example.

Suppose you're running a consumer chat assistant with the following workload:

Here's the baseline cost:

Our goal is to reduce this bill by at least 50% without sacrificing response quality. We'll measure quality using an offline evaluation dataset and monitor the thumbs-down rate in production.

Step 1: Shorten the Answers

A review of production responses reveals that many answers repeat the user's question or include unnecessary summary paragraphs.

We update the prompt instructions to encourage more direct answers and set an output token limit as a safety cap.

This reduces the average response length from 400 to 300 tokens.

The offline evaluation shows no meaningful drop in quality, and the thumbs-down rate remains stable during an A/B test.

Since input costs remain unchanged, the new monthly cost is:

That's $45,000 saved per month from a relatively simple change.

Step 2: Enable Prompt Caching

Next, we examine prompt construction and discover that the system prompt starts with the current date and the user's name. Because these values change between requests, they prevent the shared prompt prefix from being cached effectively.

We move the dynamic values after the static instructions, making the 2,000-token system prompt identical across users.

After enabling caching, usage metrics show that approximately 3,500 of the 5,000 input tokens are cached on average. This includes the shared system prompt and portions of conversation history reused across consecutive turns.

Assuming cached tokens cost 10% of the normal input price, with an additional average cost for cache writes:

Input cost drops from $0.015 to approximately $0.007 per request.

The risk is relatively low because we're reorganizing the prompt rather than removing information. We still run the evaluation suite to verify that response quality remains stable.

Caching also reduces time to first token, improving responsiveness while lowering costs.

Step 3: Route Simple Requests to a Smaller Model

Next, we analyze several thousand labeled conversations and find that around 30% of requests are straightforward, including greetings, simple follow-up questions, and basic factual queries.

These requests can often be handled by a smaller model without a noticeable difference in quality.

We introduce a lightweight classifier to route them to a model priced at:

Complex requests continue using the larger model. If a smaller model's response fails validation, we escalate it to the larger model.

Using the costs from the previous step:

Assume 70% of requests go directly to the large model, 30% go to the smaller model, and 1% of those routed to the smaller model require escalation. Each escalation incurs an additional large-model call.

Routing requires more evaluation than the previous optimizations because using a smaller model introduces the risk of lower-quality answers.

Before rollout, we compare both models on real requests across different categories. We then run the router in shadow mode, gradually enable it for production traffic, and monitor quality metrics for each route.

Different models generally don't share prompt caches. Routing can therefore change cache hit rates and the effective cost of each route. These calculations are illustrative; production estimates should use measured cache hit rates, escalation rates, and token usage for each model.

Step 4: Move Analytics to the Batch API

The nightly analytics job currently costs $60,000 per month because it uses the real-time API.

But these summaries aren't needed until the following morning, so there's no reason to pay for real-time processing.

We move the workload to a batch API offering a 50% discount.

This optimization doesn't require changing the model or summarization logic. It simply moves a delay-tolerant workload to a cheaper execution path.

Optimizations We Skipped

We also test response caching but decide against it.

Because each request includes its own conversation history, exact-match caching achieves a hit rate below 2%.

Semantic caching produces more matches, but it introduces the risk of reusing an answer that doesn't account for the current conversation. Given the limited savings and potential quality impact, we leave it out.

The Result

After applying these four optimizations, the monthly bill drops from $690,000 to approximately $292,000.

That's a reduction of approximately 58%, saving nearly $400,000 per month, while maintaining response quality in our evaluation and production tests.

The biggest saving came from prompt caching, which required relatively little engineering effort. Model routing delivered additional savings but required more careful testing and monitoring.

Three of the four optimizations didn't change the model at all. Removing unnecessary tokens, reusing repeated context, and moving background work to batch processing can deliver substantial savings with minimal quality risk.

Here are the key techniques for managing and reducing costs in GenAI systems:

Remove unnecessary context, cache repeated content, use the least expensive model that meets your quality requirements, and enforce spending limits so costs stay predictable as traffic grows.