AI application cost is not one line item called "the model." It is the sum of inference, embeddings, retrieval, re-ranking, evaluation, retries, storage, observability, and sometimes GPU capacity that sits idle between traffic spikes.

Optimizing only the visible LLM call often leaves much of the bill untouched. A production RAG or agent system can spend a large share of its budget on context assembly, repeated tool calls, validation retries, and background evaluation. This chapter breaks those costs down so cost decisions come from measured spend, not a surprising invoice.

The basic rule is simple: most hosted LLM APIs charge by tokens. A few details matter beyond that.

Input and output tokens are priced differently. Input tokens include the system prompt, user message, retrieved documents, tool definitions, and conversation history. Output tokens are the model's generated response. Output tokens usually cost several times more because generation is sequential: each new token depends on the tokens before it.

Cached input may be priced separately. Some providers discount repeated prompt prefixes. That changes the cost model for long system prompts, tool schemas, and repeated reference documents. A 6,000-token prompt that is mostly cacheable can be cheaper than a 3,000-token prompt whose beginning changes on every request.

Reasoning and multimodal features change the bill. Models with extended reasoning budgets may bill reasoning work separately or as output-like tokens, depending on the provider. Audio, images, search grounding, code execution, and region-specific processing can have their own rates. Cost estimates based only on text tokens will be wrong if your application uses those features.

Rather than memorizing provider prices, think in tiers. The actual numbers change often, so keep current prices in configuration and check the provider's pricing page before making product or architecture decisions.

When estimating cost, check whether your workload uses batch discounts, cached-input discounts, long-context surcharges, web-search charges, data-residency multipliers, or dedicated-capacity pricing.

A few patterns hold across providers. Frontier models are expensive enough that routing matters, so cheaper models should handle classification, extraction, short summaries, and other control-plane decisions when quality is good enough. Output tokens dominate many bills, especially when the application encourages verbose answers. Cached input and batch APIs can change the economics of long, repetitive prompts.

The diagram shows a typical RAG query. Most of the input tokens come from retrieved context and chat history, not from the user's question. A user may type 20 words while the system sends thousands of tokens to the model. Much of cost optimization is about closing that gap without removing context the model actually needs.

Token costs from the primary LLM are easy to see in a provider dashboard. Production systems carry several other cost centers, especially when they use RAG, tools, agents, or automated evaluation.

Embedding Generation

Every RAG query usually starts with a query embedding. That call is usually cheap, but indexing is not always trivial. If you have 100,000 chunks at 500 tokens each, the initial index processes 50 million tokens. Every document update, chunking change, or embedding-model migration reopens that bill.

Re-ranking Calls

A re-ranker is another model call on the hot path. Cross-encoder re-rankers process the query and candidate documents together, which can improve relevance but adds latency and cost. Some providers price re-ranking per search; others price it by tokens or request units. At top-k = 20 with 500-token chunks, a single re-rank step may process roughly 10K tokens, so the cost can grow faster than expected.

Evaluation and Testing

LLM-as-judge evaluation means using a model to score another model's output. Evaluating every production response can approach the cost of a second inference pipeline. Sampling a small percentage of traffic is often enough for monitoring, while full evaluation is better reserved for releases, regressions, and high-risk workflows.

Retries and Fallbacks

API calls fail. Rate limits happen. Structured outputs sometimes fail validation. Each retry is another cost event, and validation retries are easy to hide in helper libraries. A 5% retry rate raises spend by roughly 5% for that path if retries are similar in size; a poorly constrained JSON extraction pipeline can do much worse.

Vector Database Hosting

Vector storage is not free. Managed services charge for capacity, replicas, pods, serverless read units, write units, or some mixture of those. Self-hosting Qdrant, Weaviate, Milvus, or pgvector shifts the cost to compute, storage, backups, and operations. For small projects this may be negligible. For tens of millions of vectors with regional replicas, it becomes a serious line item.

The Full Picture

Vector DB cost amortized: monthly hosting cost divided by query volume.

In this example, the hidden costs double the per-query price. The LLM call itself is $0.010, but the total is $0.020 when retrieval, re-ranking, evaluation, retries, and vector storage are included. At 10,000 queries per day, that hidden $0.010 per query is about $3,000 of untracked monthly spend.

The instrumentation goal is straightforward: every API call the application makes should log its token usage and cost, tagged with enough metadata to answer questions like "which feature is most expensive?" and "which users cost the most?"

The cost tracker below wraps LLM calls:

Every model call emits a structured cost event. JSONL is convenient for a lesson because it is easy to inspect and aggregate, but in production this should go to a metrics pipeline or warehouse. The feature tag identifies which product surface generated the cost. user_id helps identify heavy usage or abuse, assuming you log it in a privacy-safe way. cached_input_tokens shows whether prompt caching is actually working. is_retry quantifies how much spend the recovery logic is consuming.

Calling the tracker:

Every call is now logged with token counts, cached-token counts, estimated cost, latency, and metadata. In a real system, keep pricing outside the code path and version it. Avoid storing raw user identifiers if your privacy policy does not allow it; hash or pseudonymize them before logging. When a billing anomaly shows up weeks later, knowing which price sheet was active at the time makes the investigation tractable.

Logging costs without reviewing them solves nothing. A small dashboard answers the recurring questions: how much are we spending per query, per feature, per user, and per day?

A sample output might look like this:

The rag_qa feature accounts for 59% of spend despite being 42% of calls, which points to retrieved context, model tier, or output length as the cost driver. user_892 costs almost twice as much per query as the average user, which could be legitimate heavy usage, abuse, or a retrieval path that expands too much context.

That kind of granularity changes the conversation from "AI is too expensive" to "classification is running on an oversized model, RAG is retrieving too many chunks, and one account needs a quota."

A dashboard shows what already happened. Budgets and alerts stop a runaway pipeline before the invoice does. The system below enforces spending limits per feature, per user, and globally, with notifications fired when usage approaches a threshold.

The pattern is a pre-flight check before each expensive call. When a feature or user has exceeded budget, the application can fall back to a cached response, route to a cheaper model, queue the request, shorten the context, reduce max_tokens, or return a graceful error. Either way, the degradation is chosen deliberately rather than discovered on the next monthly invoice.

Practical tips for setting budgets:

References