The Reference Architecture
Many GenAI systems share a common set of core components: an API layer that handles requests and streams responses, an orchestrator that builds prompts, a router that selects models, a retrieval system that provides relevant context, and guardrails that check inputs and outputs.
Understanding this common architecture gives you a starting point for GenAI system design interviews. Instead of designing each system from scratch, you can build on a shared foundation and focus on the requirements and challenges unique to each problem.
In this chapter, we'll explore the main layers of a GenAI system, what each does, and how they work together. We'll then look at the three main data paths, where latency and failures occur, and how to adapt the architecture to different use cases.
The Architecture at a Glance
Most GenAI applications are built around eight layers.
A request starts at the client, passes through the API gateway, and reaches the orchestrator, which gathers context, runs guardrail checks, and sends the prompt to the LLM gateway. The dotted lines are telemetry, such as traces, token usage, and cost, collected asynchronously so it doesn't slow down the response.
Here's a breakdown of what each layer is responsible for:
These responsibilities are logical boundaries, not necessarily separate services. In a smaller application, several layers might run within the same backend service. As the system grows, you can separate them based on scaling, reliability, and operational requirements.
Now, let's examine each layer in more detail, starting with the client and streaming layer.
The Layers in Detail
Client and Streaming Layer
LLM responses can take several seconds to generate. Instead of making users wait, most interactive GenAI applications stream responses as they're produced.
Server-Sent Events (SSE) work well for text-based chat, where the client sends a request and the server streams the response over HTTP. WebSockets are better for continuous two-way communication, such as voice assistants that support interruptions.
A stream typically carries several types of events:
- Text deltas: Generated text sent in small chunks.
- Status updates: Progress messages such as "Searching documents" or "Running tests."
- Citations: References supporting the generated answer.
- Completion and errors: Final status, token usage, or error details.
Two important considerations are cancellation and reconnection.
When a user stops generation, the server should cancel the model request where possible to avoid wasting resources. If a connection drops, the client may need to resume the response. For this, the server can store events with a response ID and sequence numbers, allowing the client to reconnect without starting over.
For longer tasks, such as an agent researching and writing a report, keeping a connection open for several minutes is impractical. A better approach is to run the task in the background, return a job ID, and let the client poll or subscribe to progress updates.
API Gateway and Auth
The API gateway handles authentication, request validation, TLS termination, and rate limiting, just as it does in traditional systems.
GenAI applications introduce two additional considerations.
Long-lived connections: Streaming responses may keep connections open for 30 seconds or longer. Load balancers and proxies need suitable timeout settings, and capacity planning must account for concurrent connections, not just requests per second.
Identity propagation: User identity, tenant information, and permissions must travel through the system. For example, when an assistant searches internal documents, the retrieval layer must know which documents the user can access.
Orchestration Layer
The orchestration layer is where most of the application's logic lives. It coordinates how the model interacts with conversation history, external data, and tools to produce a response. Its responsibilities fall into three main areas.
1. Prompt Assembly
Before calling the model, the orchestrator builds the prompt from system instructions, conversation history, retrieved documents, tool definitions, and the user's message.
All of this must fit within the model's context window, leaving enough room for the response. Larger prompts also increase cost and latency, so each component needs a token budget.
For example, a customer support assistant might allocate a 16,000-token input budget as follows:
These budgets are adjustable. If conversation history grows too large, older messages can be summarized or removed. If retrieval returns too much content, the system can select fewer, more relevant passages.
Another useful optimization is prompt caching. System instructions and tool definitions often stay the same across requests. Keeping this stable content at the beginning of the prompt allows supported model providers to reuse cached prefixes, reducing cost and latency.
2. Tool Execution Loop
For AI agents, the orchestrator also manages tool calls.
When the model requests a tool, the orchestrator validates its arguments, checks permissions, executes the tool, and sends the result back to the model. This continues until the model produces a final response or reaches an execution limit.
The model decides which tools to request, but the orchestrator controls their execution.
The orchestrator should enforce limits on tool calls, execution time, and token usage. Without these limits, an agent can get stuck in repeated tool calls, consuming resources without making progress.
Tools that perform actions, such as issuing refunds or placing orders, also need safeguards. They may require user approval, and operations should be designed to avoid duplicate actions during retries.
3. State Management
Ideally, orchestration workers should be stateless, making them easier to scale horizontally.
Persistent information, including conversation history, agent progress, and pending approvals, belongs in an external database. Workers can retrieve this state using a conversation or session ID.
Prompts should also be versioned. Even small instruction changes can affect model behavior. Recording which prompt version was used for each request helps investigate regressions and roll back problematic changes.
LLM Gateway
The LLM gateway provides a common interface between the orchestrator and different models or providers.
Its main responsibilities include:
- Model routing: Choose a model based on task complexity, cost, latency requirements, or user plan.
- Rate limiting: Enforce request and token quotas across users, tenants, and providers.
- Retries and fallbacks: Handle provider failures and rate limits, switching to alternative models when appropriate.
- Caching: Reuse identical responses, safely cache semantically similar requests where appropriate, and support provider-side prompt caching.
- Usage tracking: Record token consumption and cost by user, tenant, and feature.
For a small application using one provider, these responsibilities can live in the orchestrator. A separate gateway becomes more valuable when multiple services or teams need centralized routing, quotas, and cost management.
Model Layer
The model layer handles inference. Models can be accessed through hosted APIs or deployed on your own infrastructure.
Hosted APIs are easier to start with. Providers manage GPUs, scaling, and inference infrastructure, while you pay based on usage. The trade-offs include provider limits, less infrastructure control, and potentially higher costs at scale.
Self-hosted models provide greater control over deployment, data handling, and performance. Inference engines such as vLLM and TensorRT-LLM use optimizations like continuous batching to improve GPU utilization and throughput.
Self-hosting can reduce costs at high, predictable traffic volumes, but requires managing GPU capacity, scaling, deployments, and failures.
A GenAI application may also use multiple specialized models within a single request. For example, a RAG system might use an embedding model for search, a reranker to select relevant documents, and a larger LLM to generate the answer. Each has different latency and cost characteristics.
Retrieval Layer
The retrieval layer gives the model access to information outside its training data, such as internal documents, product catalogs, and frequently updated content.
It has two main components: ingestion and search.
Ingestion processes documents in the background. It extracts content, splits it into chunks, generates embeddings, and stores them in a search index with metadata such as source, timestamp, and access permissions.
A message queue can handle document updates asynchronously, keeping ingestion separate from user requests.
Search retrieves relevant information when a user asks a question. A common approach is hybrid search, combining:
- Vector search: Finds content with similar meaning.
- Keyword search: Finds exact terms such as product names, identifiers, and error codes.
The results are merged and passed through a reranker, which selects the most relevant passages for the model's prompt.
Search must also enforce access control, applying document permissions so users only retrieve content they're authorized to access.
Permission filtering should happen during retrieval, not just after selecting the top results. Otherwise, removing unauthorized documents may leave too few relevant passages. Asking the model to ignore restricted information is not a reliable security measure.
Guardrails
Guardrails check inputs and outputs to reduce unsafe or unwanted behavior.
Input guardrails examine user messages and untrusted content, including retrieved documents and tool results. They can detect sensitive information, policy violations, and prompt injection attempts.
Output guardrails validate generated responses. Common checks include JSON schema validation, sensitive data redaction, content policy enforcement, and citation verification.
The main trade-off is safety versus latency. Simple validation rules are fast, while checks requiring another LLM call can introduce noticeable delays.
A few techniques help reduce the overhead:
- Parallel checks: Run independent validation and retrieval operations concurrently, while ensuring rejected content never reaches the model.
- Incremental validation: Check buffered chunks of streamed output before sending them to users.
- Selective checks: Use lightweight validation by default and more expensive checks when necessary.
Guardrails reduce risk but cannot guarantee safety. For agents, permissions must also be enforced at the tool execution layer so that a manipulated model cannot access restricted data or perform unauthorized actions.
Evaluation and Observability
Traditional monitoring tells us whether a system is fast and reliable. GenAI systems also need to measure whether their responses are accurate and useful.
Observability starts with tracing. Each request produces a trace containing operations such as retrieval, model calls, guardrail checks, and tool execution.
These traces help identify where latency occurs and why a response failed. They can also record model versions, prompt versions, token usage, and costs.
Common operational metrics include time to first token, end-to-end latency, error rates, and cost per request. Traces may contain sensitive data, so they need appropriate access controls and retention policies.
Quality evaluation typically happens in two ways:
- Offline evaluation: Run predefined test cases before deploying changes to models, prompts, or retrieval pipelines. Compare results against a baseline to catch regressions.
- Online evaluation: Measure quality in production using sampled responses, automated graders, human reviews, and user feedback.
These approaches complement each other. When production monitoring reveals a recurring failure, add it to the offline evaluation set so future changes can be tested against it.
Three Paths Through the System
Now that we've covered the main layers, let's look at how data moves through a GenAI system.
Most GenAI systems have three main paths, each with different responsibilities and performance requirements.
The request path processes user queries and determines response latency and cost. The data path keeps the retrieval index up to date so the model can access current information. The feedback path uses production data to identify problems and improve response quality over time.
Let's look at the main design considerations for each.
The Request Path
Every step in the request path contributes to response latency. In system design interviews, it's useful to estimate a latency budget to identify potential bottlenecks.
For example, consider a customer support assistant targeting a time to first token (TTFT) under two seconds.
These are illustrative estimates, assuming some operations run in parallel.
The main trade-off is between response quality and latency. Query rewriting and reranking can improve retrieval accuracy but add processing time. Reducing retrieved context lowers token costs and model processing time.
Also, remember that total latency depends on the longest chain of dependent operations, not simply the sum of every step.
The Data Path
The data path runs independently of user requests. Its main responsibility is keeping the retrieval index up to date.
For example, if a company changes its refund policy but the index still contains the old version, the assistant may return outdated information even when the request path works correctly.
There are three main considerations:
- Freshness: Use event-driven ingestion or scheduled updates depending on how quickly source data changes. Reprocess only documents that have changed when possible.
- Deletions and permissions: Remove deleted documents and apply access changes promptly to prevent outdated or unauthorized content from appearing in search results.
- Re-indexing: Changes to embedding models may require rebuilding the index. Build and validate the new index in the background, then switch traffic over without interrupting queries.
The Feedback Path
The feedback path helps improve response quality using production traces, user feedback, automated evaluations, and human reviews.
For example, if users repeatedly report incorrect refund information, the team can investigate whether the problem comes from retrieval, prompt instructions, or the model.
Once the issue is fixed, the failing examples are added to the offline evaluation dataset. Future changes can then be tested against those cases to prevent the same problems from returning.
Common Failure Points
Every layer has its own failure points. Knowing them helps you answer "What happens if this component fails?"
Not every failure produces an error.
For example, if the retrieval system misses a relevant document, the model might still generate a convincing but incorrect answer. Similarly, a prompt update could reduce response quality without affecting latency or availability.
In both cases, the API might return HTTP 200, and traditional monitoring would show nothing unusual. Failures like these show up through the feedback path, in user ratings and quality evaluations.
Adapting the Architecture to Different Problems
The layers stay the same across problems, but their importance changes. Here's how the architecture shifts for different types of GenAI systems:
Let's take a coding agent as an example.
The retrieval layer now works with source files, symbols, and code dependencies instead of knowledge-base articles. The orchestration layer becomes more complex because the agent needs to plan changes, execute commands, inspect results, and retry when something fails.
We also need an additional component: a sandboxed execution environment where the agent can run code, execute tests, and modify files without unrestricted access to the host or production systems.
The API gateway, LLM gateway, and observability layer are still part of the system, but their responsibilities remain largely unchanged.
The main design challenges are now in the tool execution loop and sandbox. You need to decide how many steps the agent can take, how it recovers from failed tests, how execution state is maintained, and how to prevent it from running destructive or unauthorized commands.
Now consider a different problem: designing an LLM gateway or inference platform.
In this case, the orchestration and retrieval layers become much less important, while the LLM gateway and model-serving infrastructure become the focus of the entire design.
What was previously a single component now expands into multiple services responsible for model routing, quota management, caching, provider integrations, and potentially GPU scheduling.
Using the Reference Architecture in an Interview
In a system design interview, use the reference architecture as a checklist, not a template you need to reproduce in full. Drawing all eight layers upfront can waste time on components the problem doesn't need.
Instead, start with a simple request path and expand the design as requirements become clear.
Begin with the client, orchestrator, and model. Add retrieval if the system needs internal documents, permission filtering if users have different access levels, and an LLM gateway if you need multiple providers or centralized routing and failover. Every component should solve a specific problem you can explain.
Before the deep dive, check the three main paths:
- Request path: Can it meet the latency requirements?
- Data path: How does it handle document updates, deletions, and freshness?
- Feedback path: How will you detect and investigate declining response quality?
Not every interview problem requires all three paths in detail, but reviewing them helps you identify gaps before moving on.
Summary
The reference architecture in this chapter has eight layers and three paths. Here are the key design decisions for each layer:
And the three paths:
In a system design interview, start with the simplest architecture that meets the requirements, add components as needed, and spend the deep dive on the hardest parts of the problem.