When you call a hosted LLM API, a lot happens behind that single request. The provider schedules it on a GPU serving many concurrent requests, manages memory as tokens are generated, and streams the response back with minimal delay. When you host a model yourself, you're responsible for all of this.

How well you handle these tasks determines latency, cost, and how many requests your GPU fleet can serve. Even with the same model and hardware, throughput can vary significantly depending on how requests are batched, how GPU memory is managed, and how computation is distributed across GPUs.

In this chapter, we'll start by looking at what happens during inference and why it matters for performance. Then we'll cover the key latency metrics, optimizations used by modern inference engines, scaling across multiple GPUs, serving frameworks, and autoscaling. Finally, we'll walk through how to estimate the number of GPUs needed to handle a given workload.

Every LLM request goes through two phases: prefill and decode.

Prefill processes the input tokens in parallel in a single forward pass. For a 3,000-token prompt, the model computes attention keys and values for all input tokens and produces the first output token.

Decode generates the remaining output one token at a time. Each step uses the latest token and the cached attention state to predict the next. A 300-token response requires roughly 300 generation steps in total, including the first token produced after prefill.

During prefill, the model stores attention keys and values for the input tokens in the KV cache. Each decode step reuses this cache and adds an entry for the newly processed token, avoiding the need to recompute attention states for earlier tokens.

Both phases use the same model weights, but they stress the GPU differently.

Prefill Is Compute-Bound

Prefill is typically compute-bound because the GPU processes many input tokens in parallel. Model weights are loaded from GPU memory (HBM) and reused across those tokens, allowing the GPU to perform a large amount of computation per byte transferred.

As the input grows, prefill takes longer, increasing the time before the first token appears. For moderate context lengths, this cost grows roughly linearly with input size, although attention computation becomes more expensive at very long context lengths.

Decode Is Memory-Bandwidth-Bound

Decode behaves differently. For a single request, each step must access the model weights and the request's KV cache, but performs relatively little computation per byte loaded. As a result, decode is typically limited by memory bandwidth rather than GPU compute.

Consider an 8B model in BF16, which requires approximately 16 GB for its weights, running on an NVIDIA H100 SXM with 3.35 TB/s of memory bandwidth.

This is a theoretical upper bound for a single request, considering only the time needed to read the model weights. Actual throughput is lower because of KV cache reads, computation, and other overhead.

The main optimization is to decode multiple requests together. With a batch of 64 requests, the GPU can reuse the loaded weights to generate 64 tokens in one step, one per request. Each request still has its own KV cache, but overall throughput improves significantly until compute capacity or KV cache bandwidth becomes the bottleneck.

This distinction explains why different serving optimizations target different phases:

The balance depends on the workload. A RAG assistant processing 10,000 tokens of retrieved context to generate a 300-token answer has a relatively heavy prefill phase. A reasoning model generating 5,000 tokens from a short question spends much more of its time decoding.

Understanding this difference is important because optimizing prefill and decode requires different strategies.

A single latency number doesn't tell you much about how responsive an LLM feels. With streaming responses, users notice two things: how quickly the first token appears and how smoothly the rest of the response arrives.

We measure these using four metrics:

Optimizing one metric can hurt another. Larger batches generally improve throughput but can increase the time between tokens for individual requests. Prioritizing a long prefill reduces that request's TTFT but can delay decoding for other requests sharing the GPU.

This is why serving systems also track goodput: the number of requests completed per second while meeting their latency targets. A system that generates more tokens but consistently misses its TTFT or TPOT targets isn't necessarily performing better.

Track p95 and p99 latency alongside averages. A small number of long prompts or overloaded batches can cause noticeable delays even when average latency looks healthy.

Latency targets depend on the application:

These are illustrative targets, not universal requirements. In a system design discussion, establish TTFT and TPOT targets early, since they influence batching, scheduling, and GPU capacity decisions.

One of the main constraints on how many concurrent requests a GPU can handle is KV cache memory, which we'll look at next.

During decoding, each new token needs to attend to all previous tokens. Recomputing their attention keys and values at every step would be expensive, so the model stores them in GPU memory as a KV cache.

The memory required per token depends on the model's architecture:

The factor of 2 accounts for storing both keys and values.

For Llama 3.1 8B (32 layers, 8 KV heads, head dimension 128, BF16), this works out to 128 KiB per token, or approximately 0.5 GB for a 4,000-token sequence.

Model weights occupy a fixed amount of GPU memory, while the KV cache grows with the number of input and generated tokens. The memory available for the KV cache therefore limits how many requests can run concurrently.

Consider an 80 GB H100 serving Llama 3.1 8B. If approximately 55 GB is available for the KV cache after accounting for model weights and runtime overhead, it can hold:

These are theoretical capacity estimates that assume efficient memory allocation.

The implication is important: longer contexts consume more KV cache memory, reducing the number of requests a GPU can serve concurrently and potentially increasing cost per token.

Early serving systems made this problem worse by reserving contiguous memory for the maximum expected sequence length. A request allocated space for 2,048 output tokens might generate only 200, leaving most of that reserved memory unused. Fragmentation made it even harder to reuse the remaining GPU memory.

The vLLM team found that earlier serving systems could waste 60% to 80% of KV cache memory through fragmentation and over-reservation. Modern inference engines address this problem through more efficient memory management, allowing larger batches and better GPU utilization.

Continuous Batching

With static batching, the server groups requests and processes them until every request in the batch finishes. Since responses have different lengths, shorter requests finish early, leaving GPU capacity unused while the longest request continues. New requests must wait for the entire batch to complete.

Continuous batching solves this by updating the batch after each generation step. When a request finishes, its slot becomes available, and the scheduler can admit another request without waiting for the others.

Introduced in the Orca paper, continuous batching is now a standard optimization in modern inference engines. It keeps the GPU busy and improves throughput, especially when requests have widely varying output lengths.

However, it introduces a scheduling challenge. If a new request arrives with a 20,000-token prompt, processing its prefill can delay decoding for existing requests, causing noticeable pauses in their streamed responses.

Chunked prefill addresses this by splitting long prefills into smaller chunks that can be scheduled alongside decode steps. This helps maintain consistent TPOT, although it may increase TTFT for requests with long prompts.

PagedAttention

Continuous batching works best when GPU memory can accommodate many concurrent requests. But as we saw earlier, traditional KV cache allocation wastes memory through fragmentation and over-reservation.

PagedAttention, introduced by vLLM, addresses this using an approach similar to virtual memory in operating systems.

Instead of allocating one large, contiguous region for each request's KV cache, it divides the cache into fixed-size blocks, commonly 16 tokens each. Each request maintains a block table that maps its logical token blocks to physical blocks anywhere in GPU memory.

New blocks are allocated as the sequence grows, so requests don't need to reserve memory for their maximum possible length. The main remaining waste is unused space in the last block.

In its original evaluation, vLLM reported less than 4% KV cache memory waste and 2 to 4 times higher throughput than earlier serving systems at comparable latency. The improvement came largely from fitting more concurrent requests into GPU memory.

Block tables also support memory sharing. When multiple requests have the same prefix, they can reference the same physical KV blocks instead of storing duplicate copies.

Prefix Caching

Many LLM requests contain repeated input: the same system instructions, conversation history sent again on every turn, or tool definitions included in every agent call.

Prefix caching avoids recomputing these tokens. The inference engine retains their KV cache entries so subsequent requests with the same prefix can reuse them and perform prefill only for the new tokens.

Different engines manage this cache differently. vLLM uses hashes to identify matching token blocks, while SGLang uses a radix tree through its RadixAttention mechanism.

Consider a request containing 6,000 tokens of shared instructions and history, followed by 100 new tokens. With a cache hit, the engine can reuse the KV cache for approximately 98% of the input, significantly reducing prefill work and TTFT.

Two factors determine how effective prefix caching will be:

Cache entries also need appropriate tenant isolation to prevent unintended reuse across security boundaries.

Quantization

Quantization reduces the number of bits used to represent model weights and, in some cases, activations or KV cache entries.

Lower precision reduces memory requirements and the amount of data transferred during inference. This can make larger models fit on fewer GPUs and improve decode throughput.

However, lower precision doesn't always mean faster inference.

For example, weight-only INT4 quantization may provide little benefit during prefill. Some kernels convert weights to higher precision before performing matrix multiplications, adding overhead. At large batch sizes, where computation rather than memory bandwidth becomes the bottleneck, its decode advantage may also shrink.

FP8 works differently on supported hardware because matrix multiplications can run directly using FP8 operands, improving both memory efficiency and compute performance.

As a practical guideline, FP8 is a strong option when supported by the GPU and model, while INT4 is particularly useful when GPU memory is the main constraint.

Regardless of the format, evaluate quantized models on your own workloads. Average benchmark scores can hide quality regressions in specific tasks, such as coding, reasoning, or structured output generation.

Speculative Decoding

Autoregressive decoding generates one token at a time, leaving much of the GPU's compute capacity underused at small batch sizes.

Speculative decoding takes advantage of this by using a smaller, faster draft model to predict several tokens ahead. The larger target model then verifies these candidates in a single forward pass.

Because verification processes multiple candidate positions in parallel, it can be much cheaper than generating those tokens sequentially with the target model, particularly when inference is memory-bandwidth-bound.

Suppose the draft model proposes four tokens and the target accepts the first three. The system keeps those three, samples a correction for the fourth, and advances four tokens with one target-model verification pass plus the drafting overhead.

With the standard rejection-sampling algorithm, speculative decoding preserves the target model's original output distribution. It improves generation speed without changing the statistical behavior of the model.

Depending on the model and workload, speculative decoding can deliver 2 to 3 times faster generation, particularly for predictable outputs such as code or structured data.

There are several ways to generate draft tokens:

The main trade-off is the additional computation. If the draft model makes poor predictions, many candidates are rejected, and the extra work provides little benefit.

At high batch sizes, the target GPU may already be compute-bound, making speculative decoding less attractive and potentially reducing overall throughput.

For this reason, speculative decoding is most useful for latency-sensitive workloads at low to moderate concurrency, where faster token generation matters more than maximizing total GPU throughput.

A 70B model in BF16 requires approximately 140 GB just for its weights, more than a single 80 GB GPU can hold. Even when a model fits on one GPU, distributing it across multiple GPUs can free up memory for the KV cache and reduce inference latency.

There are several ways to distribute a model, each with different trade-offs.

Tensor parallelism splits the weight matrices within each layer across multiple GPUs. With four GPUs, each holds roughly a quarter of the weights, computes part of the result, and communicates with the others to combine the outputs. Because each GPU processes less data, tensor parallelism can reduce decode latency. However, frequent communication between GPUs requires fast interconnects like NVLink, which provides up to 900 GB/s of aggregate bandwidth on H100 SXM systems. Tensor parallelism is therefore typically used within a single 8-GPU node.

Pipeline parallelism divides the model by layers instead. For example, with an 80-layer model and four GPUs, each GPU handles 20 consecutive layers. This requires less communication between GPUs, making it suitable for scaling across nodes. However, each request still passes through every stage sequentially, so pipeline parallelism doesn't inherently reduce single-request latency. It also suffers from idle GPU time when there aren't enough concurrent requests to keep all stages busy.

Very large models often combine both approaches. A 405B model in BF16 needs approximately 810 GB for its weights, exceeding the 640 GB available in an 8×H100 node. One option is to use tensor parallelism across eight GPUs within each node and pipeline parallelism across two nodes. Alternatively, FP8 quantization reduces weight memory to approximately 405 GB, allowing the model to fit within a single node, assuming sufficient memory remains for the KV cache and runtime overhead.

Other scaling strategies serve different purposes. Expert parallelism distributes the experts of a mixture-of-experts model across GPUs. Data parallelism handles higher traffic by running independent model replicas behind a load balancer.

Separating Prefill and Decode

Even with chunked prefill, both phases still compete for resources on the same GPUs. A burst of requests with long prompts can delay ongoing decode steps, increasing TPOT for other users.

The two phases also have different hardware requirements. Prefill benefits from high compute throughput, while decode depends heavily on memory bandwidth and efficient batching.

Disaggregated serving separates them into dedicated GPU pools. A prefill worker processes the input and builds the KV cache, then transfers that cache to a decode worker, which generates and streams the remaining tokens.

This allows each pool to be optimized and scaled independently. If TTFT increases, you can add prefill capacity. If TPOT increases, you can add decode capacity without changing the prefill pool.

Systems like DistServe and Splitwise have demonstrated significant goodput improvements with this approach. Moonshot AI's Mooncake is one production example, and at the time of writing, serving frameworks such as vLLM, SGLang, and NVIDIA Dynamo support this architecture.

The trade-off is additional complexity. KV caches for long prompts can occupy several gigabytes, making transfers expensive without high-bandwidth networking. You also need to manage two GPU pools, coordinate scheduling, and handle failures during KV cache transfers.

Disaggregated serving is most useful at large scale, especially for workloads with long prompts and strict latency targets. For smaller deployments, a shared GPU pool with continuous batching and chunked prefill is usually the simpler starting point.

You rarely need to implement these serving optimizations from scratch. Modern inference engines already handle batching, KV cache management, and other performance improvements. The bigger decision is which serving stack to use and whether to self-host models at all.

For self-hosting, vLLM is a good starting point because of its broad compatibility and mature ecosystem. SGLang is particularly attractive for workloads with substantial prefix reuse, such as multi-turn conversations and agents. TensorRT-LLM is worth considering when running a specific model at high volume on NVIDIA hardware, where additional optimization effort can deliver meaningful savings.

The decision to self-host largely comes down to cost, utilization, and control.

With hosted APIs, you pay based on token usage. With self-hosting, you pay for GPU capacity whether it's being used or not, along with the cost of operating the infrastructure.

Self-hosting becomes attractive when traffic is high and predictable, you need tighter control over latency or data handling, or you want to serve custom or fine-tuned models. Hosted APIs are usually a better choice when traffic is low or unpredictable, access to frontier models matters, or the team doesn't want to manage GPU infrastructure.

A common approach is to combine both: use a self-hosted open model for simple, high-volume requests and route more complex tasks to a hosted frontier model. This lets you balance cost, quality, and operational complexity without committing to a single serving strategy.

Autoscaling an LLM service is similar to autoscaling a traditional web service, with one important difference: a new GPU replica can take several minutes to come online.

Provisioning a GPU node can take minutes, and during traffic spikes, cloud providers may not have enough GPU capacity available. Container images can exceed 10 GB, and model weights are even larger. Downloading 140 GB of weights at 1 GB/s takes at least 140 seconds, or over two minutes, before GPU loading and warm-up are complete.

A cold replica can take several minutes to become ready, depending on image size, weight size, and node provisioning. By then, a short traffic spike may already be over.

There are several ways to reduce startup time:

Choosing the right scaling metric is equally important.

Traditional autoscalers often rely on CPU or GPU utilization, but GPU utilization alone can be misleading for LLM inference. Tools like nvidia-smi report how often the GPU is executing kernels, not how efficiently its compute and memory resources are being used. Even a single decoding request can show high GPU utilization while leaving substantial capacity for batching additional requests.

Inference engines expose more useful signals:

A practical autoscaling policy adds replicas when queue depth or KV cache usage remains high for a sustained period. It scales down more conservatively, allowing active requests to finish before removing capacity.

Because GPU replicas are expensive to start and slow to replace, autoscaling should anticipate demand where possible rather than react only after latency increases.

Let's estimate the GPU capacity needed to serve a self-hosted RAG support assistant.

Assume the following requirements:

Step 1: Estimate GPU Memory

A 70B model in FP8 requires approximately 70 GB for its weights. That leaves very little memory for the KV cache on a single 80 GB H100.

Instead, we'll use two H100s with tensor parallelism of 2, giving us 160 GB of combined GPU memory.

Assuming 90% of that memory is usable after reserving space for runtime overhead:

The KV cache estimate includes both input and output tokens and represents the combined memory across both GPUs.

This gives us a theoretical limit of approximately 68 concurrent requests per replica. However, fitting that many requests into memory doesn't mean we can serve them while meeting our latency targets.

Step 2: Benchmark Against Latency SLOs

As concurrency increases, the GPU can process more requests together, improving throughput. But larger batches also increase decode time, and requests may spend longer waiting in the queue.

We need to find the highest concurrency that still meets our TTFT and TPOT targets.

Suppose load testing with realistic prompt and response lengths produces these results:

These numbers are illustrative, but the pattern is typical. Throughput initially increases with concurrency, then starts to flatten while latency continues to rise.

At 64 concurrent requests, throughput barely improves, but both latency SLOs are violated.

The best operating point is therefore 48 concurrent requests per replica, where we achieve approximately 1,140 output tokens per second while meeting both latency targets.

Step 3: Calculate Required GPUs

Now we can convert the benchmark results into requests per second.

Each request generates approximately 300 output tokens, and each replica produces 1,140 output tokens per second at our chosen operating point.

We can also sanity-check the result using Little's Law, which relates concurrency, throughput, and average request duration:

This is consistent with requests generating 300 tokens at roughly 42 ms per token. For precise capacity planning, we'd use measured average request duration rather than deriving it from p95 latency metrics.

The additional 30% capacity provides headroom for traffic spikes, replica failures, and rolling deployments.

Step 4: Estimate Cost

Assume an illustrative GPU rental price of $3 per H100-hour.

Running 36 H100s continuously costs:

That's approximately $2,600 per day at peak capacity, with opportunities to reduce costs by scaling down during quieter periods.

Now compare this with a hosted API charging $3 per million input tokens and $15 per million output tokens.

If average traffic is one-third of peak, we'd handle approximately 1.44 million requests per day.

At this scale, self-hosting appears significantly cheaper.

However, this isn't a complete cost comparison. The self-hosted estimate excludes engineering time, monitoring, infrastructure overhead, and additional capacity for high availability or multiple regions. It also assumes the open-weight model delivers sufficient quality for the application.

At lower traffic volumes, GPU utilization falls and the economics change. Hosted APIs become more attractive because you pay only for usage, while self-hosted GPUs incur costs even when idle.

There's also room to improve our estimate. Support requests often share system instructions and conversation context, making prefix caching a useful optimization. Reducing repeated prefill work could improve TTFT and allow more concurrent requests per replica.

But every major optimization changes the performance characteristics of the system, so we'd need to rerun the benchmarks before reducing the GPU count.

Interview Summary

In an interview, you could summarize the estimate like this:

Each replica uses two H100s to serve a 70B model in FP8. Memory allows approximately 68 concurrent requests, but load testing shows we can meet our latency targets at 48 concurrent requests, delivering about 3.8 RPS per replica.To handle 50 RPS at peak with 30% headroom, we need 18 replicas, or 36 H100s. I'd autoscale based on queue depth and KV cache usage, maintain warm capacity for traffic spikes, and use prefix caching to reduce repeated prefill work.

Efficient LLM serving comes down to understanding two phases with different performance bottlenecks, managing GPU memory effectively, and balancing latency against throughput.

Here are the key decisions covered in this chapter:

In an inference design discussion, identify the bottleneck before choosing an optimization.

High TTFT may indicate expensive prefill or excessive queueing, making prefix caching, additional compute capacity, or prefill disaggregation worth considering. High TPOT points toward decode performance, where batching strategies, quantization, and speculative decoding can help. If concurrency is limited by GPU memory, focus on KV cache efficiency and memory usage.

Once you understand what's limiting performance, you can choose the right optimizations and estimate the GPU capacity needed to meet your latency and throughput targets.