Many production AI failures are not caused by the model itself. They come from the system around the model: slow request paths, fragile retrieval, missing backpressure, runaway agent loops, weak observability, and unclear ownership between the application and the model layer. A strong model helps, but it cannot make up for an architecture that does not control latency, cost, state, and failure.

In this chapter, you will learn four architecture patterns that appear again and again in real AI products: request-response, pipelines, agent loops, and event-driven processing.

Traditional web applications are usually built around short, predictable operations. A database query may take a few milliseconds. An internal service call may take tens of milliseconds. You can often fit several of those operations into one user request and still respond in under a second.

LLM calls change that mental model. A short answer may come back in a second or two. A long-context answer, a reasoning-heavy request, or a multimodal call can take much longer. A RAG system may retrieve, filter, rerank, and generate before the user sees the final answer. An agent may need several tool calls before it knows whether the task is complete.

That makes architecture visible to the user. Whether a call is synchronous or asynchronous, where you place queues, whether you stream progress, and what you cache all affect how fast and reliable the product feels.

Cost is the other constraint. Every model call, embedding job, rerank, and image or audio operation consumes billable capacity somewhere. If your system reprocesses the same document, repeats the same generation, or lets an agent loop without a budget, the bug shows up on the invoice. Good AI architecture is about latency, reliability, and unit economics at the same time.

Let's walk through the four patterns you will see in most production AI systems.

This is the simplest pattern, and it is where many AI features start. A user sends a request, your system calls an LLM, and the user gets a response.

How It Works

The request-response pattern maps directly to the standard HTTP request/response cycle. The main difference is that the processing step includes an LLM call, often alongside database queries, retrieval, policy checks, or other application logic.

The flow is straightforward:

When This Pattern Fits

Request-response works well when:

A Concrete Example: AI-Powered Search

Here is a small request-response example. It receives a question, retrieves a few relevant snippets, and asks the model to answer from that context:

This pattern sits behind many chatbots, customer support tools, and AI search features. It is direct, easy to reason about, and usually the right starting point. The trade-off is that the user waits for every step in the critical path. Streaming can make the experience feel faster, but retrieval, reranking, policy checks, and generation still need to finish before the answer is complete.

Handling the Latency Problem

The most common improvement is streaming. Instead of waiting for the full answer, you send partial output as the model generates it. Streaming usually does not reduce the total work, but it makes the experience feel much better because the user sees progress quickly.

Caching is the other common improvement. If the same question, retrieved context, or prompt prefix appears often, you may be able to skip embedding, retrieval, or generation. Be careful with cached answers when the underlying knowledge changes. Also keep application-level response caching separate from provider-side prompt caching, which usually has provider-specific rules and short time-to-live windows.

The pipeline pattern is for work that moves through a set of ordered stages. Each stage takes the output of the previous stage and transforms, enriches, validates, or stores it.

The Problem It Solves

A document knowledge base is a good example. When a user uploads a PDF, the system may need to:

Each step depends on the previous one. If you run the whole process inside a user request, the user may wait minutes. If embedding generation fails on chunk 47 of 200, you also need a way to resume from that point instead of repeating the whole job.

Pipelines help by breaking the work into clear stages. Each stage can be observed, retried, and scaled on its own.

How It Works

The solid arrows show the data flow. The dotted arrows show optional queues between stages. A small pipeline can call functions directly. A production pipeline often puts queues or durable job records between stages so workers can run independently, scale out, and retry failed work without restarting the whole pipeline.

Why Queues Matter

Without queues or checkpoints, a failure in the embedding stage can fail the whole ingestion job. With queues, the chunking stage can publish work and move on. The embedding stage consumes that work at a controlled rate, retries failures, and resumes after a restart.

Queues do not make the system correct by themselves. In production, you still need idempotent writes, deduplication keys, retry limits, and dead-letter handling for messages that keep failing. Queues reduce the chance of lost work, but they do not automatically prevent duplicate processing.

This pattern is not unique to AI. Traditional data systems use it for ETL and stream processing. The difference in AI systems is that some stages are slow, expensive, or rate-limited, such as embedding generation, OCR, reranking, and LLM summarization. Scaling those stages independently can make a large difference to cost and throughput.

A Concrete Example: Document Ingestion Pipeline

Here is a simplified pipeline that processes a document through chunking, embedding, and storage. A real production version would likely add queues or durable job records between stages, but the basic shape is the same.

Pipeline Variants

Not every pipeline is a straight line. Two common variations are:

Fan-out pipeline

One stage sends data to multiple downstream stages that run in parallel. For example, after chunking a document, the system might generate embeddings and a document summary at the same time. Both results can later feed into a final storage stage.

Conditional pipeline

The path depends on the data. If the uploaded file is a scanned PDF, you run OCR. If it is already plain text, you skip OCR. If the document is long, you generate a summary. If it is short, you skip summarization.

When This Pattern Fits

Pipelines work well when:

Pipelines are usually not the right shape for interactive features where the user expects an immediate answer. For those, start with request-response, often with streaming.

The agent pattern is different from request-response and pipelines. In the first two patterns, your application defines the sequence of operations. In an agent loop, the model chooses the next action from a limited set of tools, observes the result, and decides whether to continue.

How It Works

An agent loop follows a simple cycle: observe, decide, act, repeat. The model receives the current state, chooses a tool call or final answer, the application executes the tool call, and the result is added back to the conversation as a new observation. The loop continues until the model returns a final answer or the application stops it.

From an architecture standpoint, this loop has three properties that make it harder to operate than a normal request path:

Unpredictable execution time

A request-response call might take 3 seconds. An agent might make 2 tool calls or 20. You usually cannot know the total runtime upfront, which makes timeouts and user experience harder to design.

Unpredictable cost

Every loop iteration usually involves at least one model call. An agent that takes 15 iterations costs far more than a single request-response call, and later iterations often include more context. Production agents need explicit budgets: maximum iterations, maximum tool calls, maximum tokens, wall-clock timeout, and sometimes limits for specific tools.

Accumulated context

With each iteration, the conversation history grows. The model sees previous actions and observations, so later iterations can become more expensive. Eventually, you may hit context window limits. Production agents need context management: summarize old steps, drop irrelevant observations, store durable state outside the prompt, or retrieve only the pieces needed for the next step.

A Concrete Example: Research Agent

Here is a minimal agent loop that uses tools to gather information and synthesize an answer. The important architectural pieces are the loop, the tool boundary, and the exit conditions.

The important architectural details in this code are:

When This Pattern Fits

The agent pattern works well when:

The agent pattern is not a good fit when:

The event-driven pattern changes what starts the work. Instead of a user asking for an answer directly, AI processing starts when something happens in the system: a document is uploaded, a customer email arrives, a code commit is pushed, or a sensor crosses a threshold.

How It Works

In an event-driven system, components communicate through events instead of direct calls. An event producer publishes an event to a message broker such as Kafka, RabbitMQ, AWS SNS/EventBridge, or Google Pub/Sub. One or more consumers subscribe to those events and process them independently.

The AI component is just another consumer. It listens for the events it cares about, runs its processing, and then writes results to a database or publishes another event. That processing might be a simple classification call, an embedding job, a document pipeline, or even an agent loop.

Why This Pattern Is Useful

Event-driven architecture gives you three practical benefits:

Decoupling

The upload service does not need to know that an AI summarizer exists. It publishes a document_uploaded event. You can add, remove, or update AI consumers without changing the producer, which matters when product teams and AI infrastructure teams move at different speeds.

Scalability

Each consumer can scale independently. If summarization is the bottleneck, you run more summarizer workers. If classification is cheap and low-volume, it may only need one worker. The broker helps distribute work and absorb bursts.

Resilience

If the summarizer crashes, events can wait in the broker until a consumer is available again. When the consumer restarts, it can resume from the last acknowledged event.

This only works if the details are handled correctly. You need correct acknowledgements, idempotent handlers, retry limits, and a dead-letter queue for messages that keep failing. Without those safeguards, an event-driven system can still lose work or process the same event more than once.

A Concrete Example: Auto-Tagging New Documents

Here is a small event-driven example using a Python queue to simulate a message broker. In production, you would replace this queue with Kafka, RabbitMQ, or a managed cloud service.

Combining Event-Driven with Other Patterns

Event-driven is often not the whole architecture. It is a trigger mechanism that starts one of the other patterns. A document_uploaded event might trigger a pipeline for OCR, chunking, embedding, and storage. A customer_email_received event might trigger a simple classification call and draft reply. A complex_task_created event might start an agent loop in the background.

This is what makes the pattern useful: it separates "when should AI run?" from "how should this AI work be processed?"

When This Pattern Fits

Event-driven works well when:

Event-driven is not a good fit when:

Now that you have seen all four patterns, how do you choose one? Start with three questions.

Question 1: Who initiates the AI processing?

Question 2: Are the processing steps known in advance?

Question 3: How important is latency?

Here is the same idea as a decision table:

Real-World Systems Use Multiple Patterns

Most production AI systems are not just one pattern. They combine patterns at different layers. A few examples:

A customer support platform uses event-driven processing for incoming tickets, a pipeline to enrich each ticket with customer history and sentiment, and request-response when a support agent drafts a reply in real time.

A code review tool starts from pull request events, runs an agent loop or structured workflow to inspect the changes, and uses request-response when a developer asks follow-up questions about the feedback.

A document intelligence product starts ingestion when a document is uploaded, uses a pipeline for OCR, chunking, and embedding, and uses request-response for the search and Q&A interface.

The goal is not to pick one pattern forever. The goal is to understand the boundary of each pattern, then compose them deliberately. That is how you avoid turning synchronous calls, background jobs, and agent loops into one hard-to-debug request path.

References