In a typical system design problem, such as designing a URL shortener or a news feed, each request involves a fairly predictable amount of work, and the output can usually be checked against well-defined requirements.

Now imagine adding a large language model to the system. A single request might take eight seconds, cost four cents, and produce a different response even when given the same input.

You still need the familiar building blocks like load balancers, databases, and queues. But introducing an LLM creates new challenges around latency, cost, reliability, and output quality. Understanding how to handle these challenges is a major part of GenAI system design interviews.

In this chapter, we'll look at what carries over from traditional system design, the four challenges that make GenAI systems different, the additional components these systems need, and the types of questions you can expect in GenAI system design interviews.

Most traditional system design concepts still apply to GenAI systems. A chat assistant serving ten million users still needs load balancers, API servers, databases, caching, rate limiting, and monitoring.

What changes is how these components behave when an LLM becomes part of the system.

For example, requests may keep connections open for several seconds while the model streams its response. Rate limiting needs to account for token usage, not just request counts. Traditional caching becomes less effective for free-text inputs, and monitoring needs to track prompts, model responses, and token usage alongside standard metrics.

To understand the difference, let's compare two systems: a traditional help center search and an AI assistant that answers questions using help center articles.

The traditional search system looks up relevant articles in the search index and returns a ranked list, often within milliseconds. Its results are generally predictable for the same query and index.

The AI assistant does more work. It retrieves relevant passages, builds a prompt, calls an LLM, and streams the generated answer, often with citations. It may also check inputs and outputs for safety.

Both systems use the same core infrastructure, but the AI assistant introduces additional components and new trade-offs around latency, cost, reliability, and answer quality.

Adding an LLM to a system introduces four challenges that affect how we design and operate it: unpredictable outputs, difficulty measuring quality, variable costs, and higher latency.

Let's look at each one.

1. Model Outputs Can Vary

Traditional software generally produces the same output for the same input, assuming the underlying state hasn't changed. LLMs behave differently. They generate responses one token at a time, and even identical requests can produce different results.

This creates several challenges:

The main difference is that you cannot assume a model will produce the same correct result every time. Your design needs to account for that uncertainty.

2. Measuring Output Quality Is Hard

Traditional systems often have clear expectations for correctness. If a payment service charges the wrong amount, a unit test can catch it.

With GenAI applications, correctness is harder to define.

Consider an assistant that summarizes customer support tickets. There may be several good summaries, and some responses may sound reasonable while leaving out important details or introducing incorrect information.

These problems are difficult to catch with traditional unit tests. A model can also return an incorrect answer with HTTP 200 and normal latency, so standard monitoring won't flag it.

That's why GenAI systems need an evaluation pipeline.

Before deploying a change, you test it against a dataset of representative questions and compare its performance with the current system.

Evaluation continues after deployment through user feedback, automated checks on sampled responses, and human review. New failure cases are added to the evaluation dataset to catch similar problems in future releases.

3. Costs Depend on Token Usage

Traditional API requests often consume relatively predictable amounts of resources. With LLMs, the cost depends heavily on how many input and output tokens each request uses.

Most hosted LLM APIs charge separately for input and output tokens, with output tokens often costing more.

For example, at an illustrative price of $3 per million input tokens and $15 per million output tokens:

Agent tasks can be particularly expensive because they make multiple model calls, often sending previous conversation history and tool results again at each step.

These costs add up quickly. At $0.0375 per RAG answer, one million requests per day would cost approximately $1.1 million per month in model inference alone.

To control costs, GenAI systems commonly use:

The goal is to meet the application's quality requirements while keeping inference costs manageable.

4. Model Responses Take Longer

Traditional APIs often respond in milliseconds. LLM requests can take several seconds, depending on the model, input size, output length, and server load.

LLM inference has two main phases:

Prefill: The model processes the input tokens. The time until the first output token reaches the client is called Time to First Token (TTFT), which also includes overhead such as network delays and queuing.

Decode: The model generates output tokens sequentially. Generation speed is measured in tokens per second.

For example, if the first token arrives after 600 milliseconds and the model generates 60 tokens per second, a 400-token response takes approximately:

Making users wait seven seconds for an answer isn't ideal. That's why chat applications typically stream responses, displaying tokens as they're generated instead of waiting for the complete answer.

However, streaming keeps connections open longer. At 1,000 requests per second with an average duration of seven seconds, the system would have approximately 7,000 concurrent requests.

Every additional step, such as retrieval, query rewriting, or guardrails, also adds latency. You need to decide which operations belong in the critical request path and which can run in parallel or asynchronously.

Long-running tasks, such as research agents, are often better handled as background jobs with progress updates.

GenAI applications use the same infrastructure as traditional systems, but add components for model inference, retrieval, safety, and evaluation.

Here's what a typical architecture looks like:

At the center is the orchestrator, which coordinates each request. It prepares prompts, manages conversation history, calls models, and handles tool execution for AI agents.

Let's look at the other components and what each one does.

Model Layer

The model layer handles inference. You can use hosted model APIs or run models on your own infrastructure.

Hosted APIs are easier to get started with because providers manage the GPUs, model serving, and scaling. However, you pay per token and must work within their rate limits and data-handling policies.

Self-hosted models give you more control over infrastructure, data, and costs. Tools like vLLM let you serve open-weight models on your own GPUs. This can be cost-effective at high, steady traffic volumes, but you must manage GPU capacity, scaling, and operations yourself.

A system may also use multiple models for different tasks. For example, a smaller model can handle classification and query rewriting, an embedding model supports retrieval, a reranker improves search results, and a larger model generates the final answer.

Retrieval Layer

LLMs don't automatically have access to your company's private documents or the latest information. The retrieval layer finds relevant information and provides it to the model as context. This approach is called Retrieval-Augmented Generation (RAG).

It typically has two pipelines:

LLM Gateway

The LLM gateway provides a common interface for accessing models from different providers. Instead of integrating each service directly with multiple model APIs, requests go through one gateway.

It typically handles:

Fallback models should be evaluated before deployment because different models may produce different results for the same input.

Guardrails

Guardrails help enforce safety, security, and output requirements.

Input guardrails check for disallowed content, suspicious instructions, and sensitive information. They may also inspect retrieved documents and tool results for prompt injection attempts.

Output guardrails validate generated responses. They can check JSON schemas, detect sensitive information, apply content policies, and verify citations. Checking that a citation exists is not the same as confirming that the source supports the answer.

Simple checks can run using code or lightweight classifiers, while more complex cases may require an additional model call. For streaming responses, some checks must happen before content is shown to users, adding latency.

Since guardrails cannot catch everything, agents should also operate with limited permissions. Sensitive or irreversible actions may require human approval.

Evaluation Pipeline

The evaluation pipeline measures whether the application produces accurate and useful responses.

Before deployment, it runs test cases against changes to prompts, models, or retrieval logic and compares results with a baseline to catch regressions.

After deployment, it uses user feedback, sampled responses, automated grading, and human review to identify problems.

This requires structured traces containing relevant details such as the prompt version, model, retrieved context, tool calls, generated response, and latency. Sensitive information should be protected or redacted.

New failure cases can then be added to the evaluation dataset so future releases can be tested against them.

How These Components Work Together

Consider an internal HR assistant where an employee asks:

"How many vacation days can I carry over?"

Here's how the request moves through the system:

This example highlights three important parts of the architecture:

A complete GenAI system design should account for all three. The system needs to retrieve the right information, generate responses within acceptable cost and latency limits, and provide a way to measure and improve answer quality.

GenAI system design interviews generally fall into two categories: applied system design and model-centric system design.

In an applied interview, you design a product or platform that uses AI models. The focus is on building a reliable, scalable, and cost-effective application around those models.

In a model-centric interview, you design the infrastructure used to train, fine-tune, or serve models. The focus is more on machine learning infrastructure, data processing, and efficient use of computing resources.

Here's how they compare:

You can usually identify the type of interview from the question itself.

For example, "Design an AI coding agent" is an applied system design problem. You would focus on how the agent understands a codebase, retrieves context, executes tools, and handles failures.

On the other hand, "Design distributed training for a 400-billion-parameter model" is a model-centric problem. You would discuss data pipelines, GPU clusters, parallelism strategies, checkpointing, and training efficiency.

Some questions can fall into either category. For example, designing a semantic search system could focus on building the retrieval architecture or training an embedding model. When the scope isn't clear, ask the interviewer whether model training is expected.

Let's look at the common questions in each category.

Applied System Design Questions

Applied GenAI questions generally fall into five categories. Each has its own engineering challenges, which usually determine where the design discussion goes deepest.

For example, voice agents have strict latency requirements because users expect natural, real-time conversations. The system needs to handle speech recognition, model inference, and speech synthesis with minimal delay, while also supporting interruptions.

AI agents introduce challenges around tool execution and control. A coding agent might read files, modify code, execute commands, and repeat these steps until it completes a task. The design needs to limit execution steps and spending, enforce tool permissions, and recover from failures.

Platform questions are often closer to traditional system design, with more emphasis on capacity planning. For example, a shared inference platform needs to serve multiple customers efficiently without allowing one tenant to consume all the available GPU resources. This involves scheduling, batching, autoscaling, and quota enforcement.

Model-Centric System Design Questions

Model-centric interviews focus on building and operating AI models rather than the applications that use them.

Common questions include:

These interviews typically involve more detailed estimates of compute, memory, storage, and training time.

For example, if you're asked to design a distributed training system for a 70-billion-parameter model, you might need to estimate the total training FLOPs, the number of GPUs required, and how long training would take. You would also need to account for hardware failures, checkpointing, and communication overhead between GPUs.

GenAI system design builds on traditional system design, but it adds new challenges around model behavior, output quality, cost, latency, and security. Here's how each one affects the architecture.

A GenAI design still needs to be scalable and reliable, and its AI features also need to be accurate, safe, responsive, and cost-effective.