A GenAI system design interview typically starts with an open-ended question like "Design a customer support chatbot" or "Design a coding assistant for our engineers." You may get few requirements and little information about traffic, usually in a 45 to 60 minute slot.

Without a clear approach, it's easy to spend too much time discussing APIs and infrastructure, leaving little room for retrieval quality, evaluation, and cost. In GenAI systems, these decisions are often just as important as the underlying architecture.

In this chapter, we'll walk through a six-step framework for approaching GenAI system design interviews, including how to manage your time, ask the right clarifying questions, and decide which areas deserve a deeper discussion. We'll then apply the framework to a customer support chatbot.

The framework has six steps. The first two focus on understanding the problem and defining what success looks like. The next two cover the system architecture and its most challenging components. The final two focus on evaluation, safety, scalability, cost, and trade-offs.

Here's a suggested time allocation for a typical 45 to 60 minute interview:

Treat these time budgets as rough guidelines. If the interviewer wants to explore a particular component in more detail, adjust accordingly. The goal is to keep the discussion focused while making sure you cover the important design decisions.

Start by understanding who will use the system, what it needs to do, and the expected scale. GenAI systems also require questions about response quality, model behavior, and acceptable risks. The answers can significantly influence your architecture.

Standard Questions

GenAI-Specific Questions

You won't always get precise answers to every question. When the interviewer leaves something open-ended, make a reasonable assumption and state it clearly.

For example: "I'll assume 2 million daily active users, a time to first token under 3 seconds, and citations for all factual answers."

These assumptions give you a starting point for the design and allow the interviewer to correct or refine them before you go deeper.

Traditional system design focuses on metrics like availability, latency, and throughput. GenAI systems introduce another important dimension: response quality. A system can be fast and highly available but still produce incorrect or misleading answers. We need to measure both.

Three Types of Metrics

It's also important to define guardrail metrics to make sure improving one metric doesn't come at the expense of another. For example, a chatbot optimized solely for automated resolution rate might provide incorrect answers or close conversations prematurely. Tracking customer satisfaction and whether conversations are escalated appropriately can help detect these problems.

At this stage, a brief statement is enough:

"The primary business metric is automated resolution rate. For response quality, we'll measure answer correctness and citation accuracy using an evaluation dataset. For performance, we'll target a p95 time to first token below 2 seconds and an average cost below 2 cents per conversation."

The goal at this stage is to establish a complete, end-to-end architecture without getting into detailed optimizations. Focus on three main parts of the system.

The Request Path

A typical request enters through an API gateway and reaches an orchestrator. The orchestrator prepares the prompt, retrieves relevant information when needed, applies the necessary checks, and calls the model through an LLM gateway.

As you walk through this flow, estimate the latency of each step. Suppose retrieval takes 150 ms, reranking takes 100 ms, and the model takes 600 ms to generate its first token. That's already 850 ms, excluding network delays, queuing, and other processing overhead. Adding another model call for query rewriting could add roughly 300 to 500 ms more.

These estimates help you identify potential bottlenecks and decide which operations are necessary in the request path, which can run in parallel, and which can be handled asynchronously.

The Data and Ingestion Path

For retrieval-based applications, the ingestion pipeline collects documents, parses them, splits them into chunks, generates embeddings, and stores them in a searchable index. Each document or chunk also carries metadata such as timestamps, source information, and access permissions.

This pipeline runs independently of user requests. Its design determines how quickly new information becomes searchable, while permission-aware retrieval ensures users can only access content they're authorized to see.

The Model Layer

A GenAI application may use several models for different tasks: an embedding model for semantic search, a reranker to improve retrieval relevance, a smaller model for classification or query rewriting, and a larger model for generating the final response.

Explain which models you'll use and whether they'll run through hosted APIs or on your own infrastructure. Base these decisions on latency, cost, data privacy, and expected traffic.

The request path and ingestion pipeline connect through the search index. Start with this basic architecture and introduce additional components only when the requirements justify them. For example, you might need a tool execution layer if the assistant can perform actions, or a session store to maintain conversation history.

Avoid introducing agents, complex orchestration, or multiple model calls before you've established that they're necessary. Each adds latency, cost, and potential failure points. If a single retrieval step followed by one model call can meet the requirements, start with that simpler design.

With the time split above, the deep dive gets about 15 to 20 minutes, so you need to focus on the parts of the system that matter most rather than trying to cover every component.

How to Choose What to Deep Dive Into

In a GenAI system, the deep dive often falls into one of four areas:

The problem statement often gives you a clue about where to focus. Answering questions across 50 million internal documents is primarily a retrieval challenge. Serving a chatbot to 10 million users at low cost calls for a deeper discussion of model serving. A coding agent that fixes failing tests requires careful agent orchestration, while generating medical visit summaries puts more emphasis on evaluation and safety.

Before diving in, explain which area you consider most important and why. Confirm with the interviewer that this is where they'd like you to focus.

How to Structure the Deep Dive

Instead of listing technologies and their features, start with a simple design and improve it as you identify limitations.

Let's use retrieval as an example:

You can repeat this process for two or three related challenges. For example, in a retrieval deep dive, you might first improve search relevance, then address document-level permissions, and finally explain how to keep the index updated as documents change.

Keep the discussion focused on the challenges specific to the problem. Avoid spending valuable time on generic infrastructure like databases and load balancers unless they're central to the design.

Evaluation helps you determine whether changes to the model, prompts, or retrieval pipeline actually improve response quality. Safety controls help prevent harmful responses, sensitive data exposure, and unintended actions.

Evaluation

Safety

Focus on the risks that matter most for the system you're designing:

The final step is to check whether your design can handle the expected traffic, meet performance requirements, and stay within the budget defined in Step 1.

Cost Estimation and Optimization

Start by estimating the cost per request, then multiply it by daily traffic.

Suppose each request processes 6,000 input tokens and generates 300 output tokens. Using illustrative pricing of $3 per million input tokens and $15 per million output tokens:

At 1 million requests per day, that's approximately $22,500 per day in model inference costs alone, excluding retrieval, additional model calls, and infrastructure.

If this exceeds the budget, discuss how you could reduce costs:

Scaling and Failure Handling

Unlike traditional APIs, GenAI requests may keep connections open for several seconds while the model streams its response. This means you need to account for concurrent connections and token throughput, not just requests per second.

Also plan for provider outages, rate limits, timeouts, and overloaded inference servers. Use request queues, concurrency limits, timeouts, and fallback models to keep the system reliable under load.

Fallback models may differ in response quality, latency, and supported features, so evaluate them before using them in production.

Design Trade-offs

Finally, summarize the most important trade-offs in your design and explain what would make you reconsider them.

For example: "I chose a hosted LLM API because it lets us launch quickly without managing GPU infrastructure. If traffic grows significantly and becomes predictable, or if stricter data residency requirements emerge, we could consider self-hosting an open-weight model."

This shows that your design is practical to operate, scale, and maintain as requirements change.

Let's apply the six-step framework to a common interview question: "Design an AI-powered customer support chatbot."

We'll walk through a simplified version of the interview in about 10 minutes, focusing on the key decisions at each step.

1. Clarify Requirements

After discussing the requirements with the interviewer, we establish the following:

2. Define Success

We define the following metrics to measure whether the chatbot meets its goals:

3. High-Level Design

We start with a retrieval-augmented generation (RAG) architecture.

When a customer sends a message, it passes through an API gateway to the orchestrator. The orchestrator loads the conversation history, retrieves relevant documentation using hybrid search and reranking, and looks up the customer's subscription plan through a read-only tool when needed.

It then builds a prompt with the relevant context and sends it to a hosted model through an LLM gateway.

Before streaming generated content to the customer, the system checks its citations against the retrieved sources. To support streaming, it validates buffered segments of the response before releasing them. This introduces some latency, which we need to account for.

Separately, a nightly ingestion pipeline processes updated articles and refreshes the search index.

If retrieval fails to find sufficient information or the generated answer cannot be supported by the sources, the orchestrator escalates the conversation to a human agent. It also provides a summary of the conversation and relevant context so the customer doesn't have to repeat everything.

4. Deep Dive

We choose retrieval quality for the deep dive because the chatbot's ability to provide accurate answers depends heavily on finding the right documentation.

We start with vector search over document chunks. However, customers often include exact error codes, API names, or product identifiers in their questions, which semantic search alone may miss.

To address this, we combine keyword and vector search, merge their results, and use a reranker to select the 5 most relevant passages from the top 30 candidates.

We also tag articles with product versions and applicable subscription plans. This allows retrieval to prioritize documentation relevant to the customer's product version and filter out content they aren't entitled to access.

5. Evaluation and Safety

Before deployment, every significant model, prompt, or retrieval change is tested against the 500-question evaluation dataset. We use an LLM judge to assess answer correctness and source support, with its scores regularly checked against human-reviewed examples.

In production, we monitor customer feedback and review conversations with negative ratings, incorrect answers, or escalations. Recurring failures are added to the evaluation dataset.

For safety, the chatbot has read-only access to subscription data, with authorization enforced for each lookup. Sensitive information is also redacted from logs and traces.

6. Scale, Cost, and Trade-offs

With 50,000 conversations per day and an average of 3 turns each, we expect approximately 150,000 model calls per day, assuming one generation call per turn.

That's about 1.7 requests per second on average. Assuming peak traffic is 3 to 6 times higher, we should plan for roughly 5 to 10 requests per second.

Now let's estimate inference costs.

Suppose each turn processes 6,000 input tokens and generates 300 output tokens. Using the same illustrative prices of $3 per million input tokens and $15 per million output tokens:

That's above our target of 5 cents per conversation, even before accounting for retrieval and infrastructure costs.

To bring costs down, we consider two optimizations:

We would measure the savings and verify that these changes don't reduce answer quality.

Finally, we discuss the main trade-off: reranking adds roughly 100 ms to each request, but we accept that overhead because better retrieval can improve answer accuracy and reduce unnecessary escalations.

Even in this shortened example, each major design decision connects back to a requirement established in Step 1.

The discussion may not follow these steps in order, so use the framework as a guide rather than a strict checklist.

Base each design decision on the requirements, spend most of your time on the hardest parts of the system, and explain how you'll measure and maintain quality, performance, reliability, and cost in production.