Suppose you're asked to design a chat assistant serving 10 million daily users. Before choosing a model or designing the architecture, a few estimates can help you understand the scale of the system: how many tokens it needs to process per second, how much a hosted LLM API would cost, and how many GPUs you might need if you host the model yourself.

In traditional system design, capacity estimation focuses on requests per second, storage, and bandwidth. These still matter, but GenAI systems introduce another important factor: token usage, which directly affects cost, latency, and compute requirements. A request with 10,000 input tokens requires significantly more processing than one with just 500, even though both count as a single request.

In this chapter, we'll work through these estimates step by step. We'll start by estimating token counts and throughput, then calculate API costs, GPU memory requirements, and inference capacity for self-hosted models. We'll also estimate storage requirements for embeddings, conversations, and traces.

Throughout the chapter, we'll use three practical examples: a chat assistant serving 10 million daily users, a 70B parameter model using FP16 and INT4 precision, and a vector index containing embeddings for 100 million documents.

Most GenAI capacity estimates start with two questions: how many requests does the system receive, and how many tokens does each request process? From there, the calculations depend on whether we're using a hosted LLM API or running the model ourselves.

Requests per second and tokens per request help us estimate the system's total token throughput. For a hosted API, we can combine token usage with the provider's pricing to estimate daily costs. For self-hosted models, we also need to estimate GPU memory requirements, inference throughput, and the number of GPUs needed to handle the traffic.

Each estimate should inform an architectural decision.

In a system design interview, you don't need exact numbers. Use reasonable assumptions, round where appropriate, and focus on getting the order of magnitude right. A 20% difference in an estimate may not affect your architecture, but a 10x difference could change the design entirely.

LLMs process text as tokens rather than individual words or characters. Before text reaches the model, a tokenizer splits it into smaller pieces. Common words may fit into a single token, while uncommon words, numbers, and code may be split into several tokens.

Converting Text to Tokens

For English text, two simple rules of thumb are useful for estimation:

These approximations work reasonably well for English prose, but token counts vary with the content. Code and JSON contain symbols, punctuation, and identifiers that can increase token usage, sometimes bringing the average closer to 3 characters per token. Non-English text, particularly languages using non-Latin scripts, may also require more tokens to represent the same amount of information.

Using these approximations, we can estimate the size of common inputs:

These are rough estimates. Actual token counts depend on the content, formatting, and tokenizer. For example, a PDF containing mostly images or tables may have a very different token count from one containing only text.

Input and Output Tokens

An LLM request typically involves two types of tokens: input tokens, which the model processes, and output tokens, which it generates. We need to estimate them separately because they affect cost, latency, and capacity differently.

Input tokens include much more than the user's message. In a RAG system, for example, the application typically combines system instructions, conversation history, retrieved documents, and the user's question into a single prompt.

In this example, the user's question contains just 50 tokens, but the model receives approximately 10,000 input tokens once we include the additional context. If the model generates a 500-token response, the input is roughly 20 times larger than the output.

This is common in GenAI applications, and the distinction matters for three reasons:

The ratio between input and output tokens also depends on the workload. A document summarizer might process tens of thousands of input tokens to generate a short summary, while a coding agent that writes entire files could generate thousands of output tokens.

Here are some reasonable starting estimates for common workloads:

Reasoning models introduce another factor. Some models generate additional internal reasoning tokens before producing the final response. These tokens may not be visible to the user, but they still consume compute and are typically billed as output tokens. For complex tasks, they can add thousands of tokens to a single request, increasing both cost and latency.

Now that we can estimate the input and output tokens per request, the next step is to combine these numbers with expected traffic to calculate how many tokens the system needs to process per second.

Estimating traffic for a GenAI system starts the same way as traditional system design. We calculate requests per second (RPS), then multiply by the average number of tokens per request to estimate token throughput.

Using three times the average RPS for peak traffic is a reasonable starting assumption. The actual multiplier depends on usage patterns. Consumer applications may experience sharp spikes during peak hours, while internal tools used primarily during business hours can have even more concentrated traffic.

Worked Example 1: A Chat Assistant for 10 Million Daily Users

Suppose we're designing a chat assistant with 10 million daily active users. Each user sends an average of 20 messages per day, and each request processes 1,000 tokens: 800 input tokens and 200 output tokens.

Let's calculate the expected traffic and token usage:

The system needs to process approximately 1.9 million input tokens per second on average, or 110 million input tokens per minute.

This raises an important capacity question. Hosted LLM APIs typically enforce rate limits on token usage, often measured in tokens per minute. At this scale, our traffic could exceed the default limits offered by many providers.

We may need higher negotiated rate limits, provisioned throughput, or the ability to distribute traffic across multiple deployments and providers. This shows how a simple capacity estimate can influence architectural decisions.

Our estimate of 800 input tokens per request assumes the prompt size stays relatively constant. However, in a real chat application, each new request often includes previous messages, so the input grows as the conversation continues.

Suppose we have a 500-token system prompt, and each conversation turn adds 250 tokens: a 50-token user message and a 200-token assistant response.

If we include the full conversation history in every request, the input size grows like this:

For a 20-turn conversation, the average input is approximately 2,900 tokens per request, more than 3.5 times our original estimate of 800 tokens.

When every request includes the full conversation history, the total number of input tokens processed across a conversation grows roughly with the square of the number of turns.

This is why chat applications often limit how much conversation history they include in each request. They might truncate older messages, summarize previous exchanges, or use a combination of both to keep the context manageable.

The growing context also has a significant impact on API costs, which we'll estimate next.

Hosted LLM APIs typically charge based on the number of input and output tokens processed, with prices quoted per million tokens. We can estimate the cost of each request using:

API prices vary significantly across models and change over time, so we'll use illustrative prices for our calculations. Suppose a model costs $3 per million input tokens and $15 per million output tokens.

Continuing our chat assistant example with 200 million requests per day, 800 input tokens, and 200 output tokens per request:

Output tokens make up only 20% of the total tokens but account for more than half the cost, because they're priced five times higher.

Now consider our earlier estimate of 2,900 average input tokens when we include conversation history. At that size, the input cost increases to $0.0087 per request, bringing the total to $0.0117. Our daily cost jumps from $1.08 million to approximately $2.34 million.

One of the most useful metrics here is cost per user per month, which we can compare directly against subscription revenue. Our original estimate of $3.24 assumes each user sends 20 messages every day. For a free-tier product, that level of spending per active user may be difficult to sustain.

One option is to route simpler requests to a cheaper model. Suppose a smaller model costs $0.15 per million input tokens and $0.60 per million output tokens. Using our original estimate of 800 input tokens and 200 output tokens, the cost drops to $0.00024 per request, or about $0.14 per user per month.

That's more than 20 times cheaper. At this scale, model routing is a major cost lever.

Prompt Caching

Another way to reduce costs is prompt caching. Many LLM providers can reuse previously computed information for repeated prompt prefixes. When a new request begins with the same tokens as an earlier request, the provider may reuse the cached computation instead of processing those tokens again.

Cached input tokens are often billed at a discounted rate, sometimes as low as 10% of the regular input price. Caching can also reduce the time to the first output token.

Chat applications are particularly well suited for this because each new message often includes the same system instructions and conversation history from previous turns.

Let's assume that 80% of our input tokens are served from cache at 10% of the regular price. Using our revised estimate of 2,900 input tokens per request:

With these assumptions, caching reduces our estimated daily cost from $2.34 million to approximately $1.1 million, a reduction of more than 50%. That brings us close to our original estimate, even after accounting for growing conversation history.

Once caching reduces input costs, output tokens account for a larger share of the bill. At that point, other optimizations become more valuable, such as limiting unnecessarily long responses, routing simpler requests to smaller models, and avoiding redundant model calls.

However, prompt caching has some practical limitations.

Most providers match exact token prefixes, so prompt structure matters. Reusable content, such as system instructions and tool definitions, should appear at the beginning of the prompt, with frequently changing content placed toward the end.

Providers may also require a minimum cacheable prefix length, sometimes around 1,000 tokens. Cached entries typically expire after a period of inactivity, and some providers charge separately for writing to the cache. Actual savings depend on cache hit rates, prompt structure, and the provider's pricing model.

Work that doesn't need an immediate response, such as nightly summarization or evaluation runs, can also use batch APIs, which many providers offer at around 50% off in exchange for a completion window of up to 24 hours. Separating interactive requests from background jobs lets each run at the right price and latency.

So far, we've estimated the cost of using a hosted LLM API. But what if we decide to run the model ourselves?

In that case, instead of paying per token, we need to estimate how many GPUs are required to handle the traffic and how much they'll cost to operate. The first step is understanding how much GPU memory the model needs.

When running an LLM on our own GPUs, two of the biggest consumers of GPU memory are model weights and the KV cache. Model weights occupy a relatively fixed amount of memory, while the KV cache grows with the number of active requests and the length of their sequences.

Weight Memory

The memory required to store a model's weights depends on the number of parameters and the precision used to represent them.

These are theoretical minimums. Quantized models also need memory for scaling factors and metadata, and some layers may remain in higher precision. For example, a 70B model quantized to INT4 might require around 40 GB rather than exactly 35 GB.

For mixture-of-experts (MoE) models, use the total parameter count, not the active count, since all experts stay in GPU memory even though each token uses only a few.

KV Cache Memory

Model weights are only part of the memory requirement. During text generation, the model also needs to keep track of previously processed tokens.

In a standard autoregressive transformer, each new token attends to earlier tokens in the sequence. Instead of recomputing the keys and values for those tokens at every generation step, the model stores them in memory. This is called the KV cache.

For a transformer using standard multi-head or grouped-query attention, we can estimate the KV cache memory per token using:

The factor of 2 accounts for storing both keys and values.

Let's consider a 70B-class model with 80 transformer layers, 8 KV heads, and a head dimension of 128, using FP16 for the KV cache.

A single request with 128,000 cached tokens could consume around 42 GB of GPU memory just for its KV cache. That's more than the memory needed to store the model's weights in INT4.

Use the model's actual KV head count, not its query head count. Our example model has 64 query heads but only 8 KV heads because it uses grouped-query attention (GQA), where query heads share keys and values. Without that sharing, the cache would be eight times larger, about 2.6 MB per token. Storing the KV cache in FP8, where the inference engine supports it, roughly halves it again.

How Memory Limits Concurrency

Once we load the model weights into GPU memory, the remaining memory can be used for the KV cache and other runtime requirements.

We can use this to estimate how many sequences the GPUs can hold concurrently.

The 0.9 factor leaves about 10% of GPU memory for activations and other runtime needs. For tokens per sequence, a typical sequence length works for an initial estimate, since modern inference engines allocate KV cache memory as sequences grow rather than reserving the full context window up front.

Worked Example 2: A 70B Model in FP16 vs. INT4

Let's compare running a 70B model in FP16 and INT4 on NVIDIA H100 GPUs, each with 80 GB of memory.

Assume each request contains 1,500 input tokens and generates 500 output tokens. By the end of the request, the sequence contains 2,000 tokens.

Using our earlier KV cache calculation:

First, let's consider FP16.

The model weights require approximately 140 GB, so they cannot fit on a single 80 GB GPU. Two H100s would hold the weights but leave only about 4 GB for the KV cache, so let's use four.

With four H100 GPUs, we have 320 GB of total memory. After accounting for model weights and runtime overhead, approximately 148 GB remains for the KV cache.

With 148 GB available for the KV cache, we can theoretically hold around 220 concurrent sequences of 2,000 tokens.

Now let's consider INT4.

Quantizing the model to INT4 reduces its weight memory to approximately 40 GB. That means the entire model can fit on a single H100 GPU, leaving around 32 GB for the KV cache after reserving runtime overhead.

Here's how the configurations compare:

Once the weights fit, every additional gigabyte of GPU memory goes to keeping more requests in flight. INT4 lets the model fit on one GPU instead of two or four, though quantization can affect output quality and should be evaluated before adopting it.

However, fitting more requests into memory doesn't mean the GPUs can process them fast enough. GPU memory determines how many sequences we can keep active, while compute capacity and memory bandwidth determine how quickly we can process their tokens.

To estimate how much traffic our GPUs can actually handle, we need to look at inference throughput next.

LLM inference has two phases that use the GPU differently. Prefill processes all the input tokens in parallel, so it's limited mainly by compute and determines the time to first token. Decode generates output tokens one at a time. At each step the GPU reads the model weights from memory to produce one token per sequence, so decode is limited mainly by memory bandwidth and determines how fast the response streams.

Batching makes decode efficient. When the GPU generates the next token for many sequences in the same step, each weight read is shared across the batch and total throughput rises. The trade-off is that each individual sequence may stream more slowly.

For a quick capacity estimate, we can start with the compute ceiling. Processing one token through a dense transformer takes roughly two floating-point operations per parameter, so a 70B model needs about 140 GFLOPs per token. An NVIDIA H100 SXM provides a theoretical peak of around 989 TFLOPS for dense BF16, and we'll assume we reach 50% of that:

This is an upper bound. It ignores attention and communication overhead, and decode often falls short of it because of memory bandwidth. Weight-only INT4 quantization doesn't raise this ceiling either, since many implementations still run the matrix multiplications in FP16 or BF16.

Memory and compute set two separate limits on a replica. Memory caps how many requests can be in flight at once, while compute caps how many tokens per second the replica can process. Capacity is whichever limit we hit first.

For our four-GPU FP16 replica with 2,000-token requests, compute allows about 14,000 / 2,000 = 7 requests per second. If each sequence streams at around 40 tokens per second, a 500-token response takes about 13 seconds, so 7 requests per second keeps roughly 90 sequences in flight. That's well under the 220 that memory allows, so compute is the limit for this workload. Workloads with long outputs, such as reasoning models generating thousands of tokens, hold each sequence in memory far longer and can make memory the tighter limit instead.

These numbers are a starting point. Production sizing needs benchmarks with representative input lengths, output lengths, and concurrency, measuring throughput along with time to first token and tail latency.

From Replicas to Cost

Once we estimate how many requests a replica can handle, we can calculate how many replicas are needed for peak traffic.

GPU prices vary depending on the cloud provider, hardware availability, and pricing commitment. Let's use an illustrative price of $3 per H100-hour.

Our FP16 replica uses four H100 GPUs, so its rental cost is $12 per hour.

At our earlier compute ceiling of approximately 14,000 tokens per second, it could theoretically process about 50 million tokens per hour.

That's approximately $0.24 per million tokens, but only at full throughput. If the GPUs average 30% of that, the effective cost rises to about $0.79 per million tokens.

This is one of the biggest differences between hosted APIs and self-hosted inference. With a hosted API, we pay for the tokens we process. With self-hosting, we pay for GPU capacity whether it's serving requests or sitting idle, and since new replicas take time to load their weights, we need to keep spare capacity for traffic spikes.

Let's see what this means for our original chat assistant serving 10 million daily users.

Each request processes 1,000 tokens: 800 input and 200 output. Using our simplified compute estimate, a four-GPU replica has an upper-bound capacity of approximately 14 requests per second.

With peak traffic of 7,000 RPS:

That's approximately $180,000 per day in GPU rental costs, compared with $1.08 million per day using our illustrative hosted API pricing.

The two numbers aren't directly comparable. The self-hosted figure assumes we reach the estimated throughput while meeting latency targets, leaves out engineering and infrastructure costs, and a 70B open-weight model may not match the hosted model's quality. Treat it as an optimistic planning estimate.

Still, the calculation shows that at high, sustained traffic, self-hosting can be much cheaper than paying per token. The hard part is keeping GPU utilization high while meeting latency and reliability requirements, which is why some large applications combine hosted and self-hosted models and route requests between them.

Storage is generally less expensive than compute in GenAI systems, but two areas can grow quickly: vector indexes and LLM traces. Both can grow faster than expected, especially when we're handling millions of documents or requests.

Embeddings

The storage required for embeddings depends on three factors: the number of vectors, the dimensions in each vector, and the number of bytes used per dimension.

But the vectors themselves are only part of the storage requirement. Vector databases may also maintain index structures, such as an HNSW graph, to speed up similarity search. We also need to store the original text chunks and their metadata so we can retrieve them after finding relevant vectors.

Worked Example 3: Embedding 100 Million Documents

Suppose we're building a RAG system with 100 million documents, each containing an average of 1,500 tokens.

We split each document into 500-token chunks without overlap, generate a 1,024-dimensional embedding for each chunk, and store the embeddings as 32-bit floating-point numbers.

Let's estimate the storage:

That's approximately 1.2 TB for the embeddings alone, plus storage for the search index, text chunks, and metadata.

For fast similarity search, HNSW-based systems often benefit from keeping vectors and index structures in RAM. At this scale, holding more than a terabyte of vector data in memory may require distributing the index across multiple machines.

One way to reduce memory and storage requirements is vector quantization, which represents embeddings using fewer bits per dimension.

The savings are substantial. Moving from float32 to int8 reduces raw vector storage by 75%, while binary quantization can reduce it even further.

The trade-off is that compression can reduce retrieval accuracy. One approach is to search using compressed vectors in memory, then rerank the top candidates using full-precision vectors stored separately.

Finally, it's worth estimating what it takes to re-embed the whole collection, which we'd need to do when switching embedding models. That's 100M documents × 1,500 tokens, or 150 billion tokens. At an illustrative $0.02 per million tokens, the API bill is only about $3,000, but the bigger constraint is time. At a rate limit of 10 million tokens per minute, the job would take more than 10 days. Since vectors from different models can't be mixed in one index, we'd also need to build the new index alongside the old one, temporarily doubling index storage.

Conversation History and Traces

Of the two, conversation history takes less space. For our chat assistant, storing about 1.5 KB per turn (message, response, and metadata) across 200 million turns a day comes to roughly 300 GB per day, or about 110 TB per year before compression.

LLM traces can consume significantly more storage.

A trace records what happens during a model request. Depending on our observability requirements, it may include the assembled prompt, generated response, retrieved passages, tool calls, token usage, latency, and errors.

For a simple request with 800 input tokens, let's assume a trace occupies approximately 5 KB. A RAG request containing 10,000 input tokens might produce a trace closer to 40 KB.

At 200 million requests per day:

At this scale, traces can be one of the larger storage costs in the system. Sampling routine successful requests, storing references instead of repeated prompt content, and applying retention policies keep them manageable.

In a system design interview, estimation should be a short, focused part of the discussion. You don't need to calculate everything in detail. Focus on the numbers that influence your architectural decisions.

A practical approach is to follow these steps:

The most important part is explaining how these estimates affect your design.

For example, here's how you might summarize our chat assistant estimates in an interview:

"With 10 million daily users sending 20 messages each, we'd handle around 200 million requests per day, with approximately 7,000 RPS at peak. At our assumed hosted API prices, that could cost over $1 million per day. Conversation history would increase input token usage and costs, so I'd use prompt caching and summarize older messages to manage context size. For the free tier, I'd route simpler requests to a smaller model and reserve the more expensive model for complex queries or paid users."

Each number in that summary supports a decision about capacity, caching, model selection, or cost.

Estimating capacity for a GenAI system starts with two things: how many requests the system receives and how many tokens each request processes. From there, we can estimate API costs, GPU requirements for self-hosted models, and storage needs for embeddings, conversations, and traces.

Here's a quick reference for the key formulas and rules of thumb:

The estimates don't need to be precise. Use them to find the main capacity constraints and decide how the architecture should handle them.