A language model doesn't automatically have access to your company's internal wiki, last week's pricing changes, or a support ticket filed this morning. Without that information, it may admit it doesn't know the answer or generate a convincing but incorrect response.

Retrieval-augmented generation (RAG) addresses this problem by retrieving relevant information at request time and including it in the model's prompt. This allows the model to ground its answers in information beyond its training data.

The concept is straightforward, but building a reliable RAG system involves several important decisions: how to split documents into chunks, which embedding model to use, how to retrieve the most relevant information, how to keep the index up to date, and how to enforce access controls.

In this chapter, we'll walk through the two main pipelines of a RAG system, the design decisions at each stage, how to handle data freshness and access control, common retrieval failures, and when using a long context window might be a better choice than retrieval.

A RAG system has two main pipelines that work with a shared index.

The ingestion pipeline runs in the background. It collects documents from source systems, extracts their content, splits it into chunks, generates embeddings, and stores the chunks and their metadata in a searchable index.

The query pipeline runs when a user asks a question. It may rewrite the query to improve search results, retrieve relevant chunks from the index, rerank them, and pass the most useful context to the language model to generate an answer.

The two pipelines have different performance requirements. Ingestion focuses on processing large volumes of documents accurately and efficiently. A full ingestion job might take hours, while incremental updates can run continuously. The query pipeline is latency-sensitive, with retrieval and reranking typically expected to finish within a few hundred milliseconds before the model begins generating a response.

Retrieval quality depends heavily on what happens during ingestion. If a table loses its structure during parsing or important context gets separated during chunking, even a good search algorithm may struggle to find the right information. That's why we'll start with the ingestion pipeline.

Parsing

Parsing converts source documents into clean, structured text that can be indexed and searched. Markdown and HTML are relatively easy to process. PDFs, slide decks, and scanned documents are more challenging.

For example, PDFs often store text as characters positioned on a page rather than as structured paragraphs and tables. A basic text extractor might mix text from two columns, merge table cells, or include repeated headers and footers. Scanned documents require OCR, which introduces additional errors.

To handle these cases, you can use layout-aware parsers or vision models for complex documents. Tables can be converted to Markdown or HTML to preserve their rows and columns.

Inspect a sample of parsed documents before moving forward. Problems that appear to be caused by poor embeddings or retrieval often originate during parsing.

Parsing should also extract metadata, including the document's source, title, URL, last modified time, and access permissions. This information becomes important later for filtering, keeping content up to date, and enforcing access control.

Chunking

Searching smaller, focused passages is generally more effective than searching entire documents. Embedding models also have input limits, so we need to split documents into manageable chunks.

There are four common chunking strategies:

A good starting point for most document collections is structure-aware chunking with a recursive fallback. First, split documents along natural boundaries such as headings and sections. If a section is too large, split it further into paragraphs or sentences.

Chunk size involves a trade-off between retrieval precision and context. Smaller chunks (100 to 200 tokens) can match specific questions more precisely but may leave out information needed to answer them. Larger chunks (1,000 tokens or more) preserve more context, but their embeddings must represent multiple ideas, which can make retrieval less precise.

A practical starting point is 300 to 600 tokens per chunk, then adjusting based on retrieval quality.

Chunk overlap helps preserve context across boundaries. For example, with 10 to 15 percent overlap, the end of one chunk is repeated at the beginning of the next. This reduces the chance of important information being separated across chunks, though overlap is less necessary when splitting at natural section boundaries.

Two additional techniques help preserve context:

Once documents are parsed and split into chunks, the next step is making those chunks searchable by meaning.

Choosing an Embedding Model

An embedding model converts text into numerical vectors, placing text with similar meanings close together in vector space. At query time, we embed the user's question using the same model and search for the closest chunk vectors.

Benchmarks like MTEB are a useful starting point, but performance varies by domain. A model that works well on general text may struggle with legal documents or source code. Evaluate a few candidates using real questions and documents from your application.

Also consider the model's maximum input length. Text that exceeds this limit may be truncated or rejected, potentially losing important information. Finally, choose between a hosted API and a self-hosted model based on data sensitivity, traffic volume, cost, and operational complexity.

Embedding dimensions commonly range from 384 to 3,072 and directly affect storage and memory requirements. For 10 million chunks stored as 32-bit floating-point vectors:

These estimates cover only the raw vectors, excluding index overhead, metadata, and replicas. Quantizing vectors from 32-bit floats to 8-bit values can reduce vector storage by roughly four times, potentially at the cost of some retrieval accuracy.

Re-embedding When the Model Changes

Embeddings from different models generally aren't compatible because they represent text in different vector spaces. Switching embedding models therefore requires generating new embeddings for the entire corpus.

A safe approach is a blue-green migration. Build a new index in the background while the existing index continues serving queries. During the migration, write new and updated documents to both indexes. Once the new index is ready, compare retrieval quality using a labeled evaluation set, then gradually shift query traffic to it.

This is also why you should retain the original chunk text and embedding model version. Without the text, changing models would require parsing and chunking the source documents again.

Vector Index: HNSW vs. IVF

Comparing a query against every stored vector becomes expensive as the dataset grows. Vector databases address this using approximate nearest neighbor (ANN) indexes, which trade a small amount of retrieval accuracy for much faster searches.

Two common approaches are HNSW and IVF.

HNSW (Hierarchical Navigable Small World) organizes vectors into a multi-layer graph, with each vector connected to nearby neighbors. Search begins at a sparse upper layer, navigates toward the query vector, and moves through denser layers to find closer matches.

IVF (Inverted File Index) divides vectors into clusters, typically using k-means. Instead of searching the entire dataset, a query searches only the closest clusters. The nprobe parameter controls how many clusters are searched.

HNSW is a good default for many applications because it provides high recall with low query latency. For very large datasets, especially billions of vectors, IVF-PQ or disk-based indexes can reduce memory requirements and infrastructure costs.

Both approaches also provide parameters for balancing speed and accuracy. Increasing ef_search in HNSW or nprobe in IVF generally improves recall but increases query latency. Tune these parameters using a representative evaluation set rather than relying on default values.

Metadata Filtering

Many queries shouldn't search the entire index. You may need to restrict results to a particular product, tenant, date range, or documents the user is authorized to access.

There are three common ways to combine metadata filters with vector search:

For multi-tenant applications, tenant isolation is especially important. Large tenants may have dedicated indexes or partitions, while smaller tenants can share an index with tenant-level filters.

These filters must be enforced by the retrieval system, not left to the language model. A document the user cannot access should never be included in the retrieved context.

Vector search is effective at finding text with similar meanings. But not every query is about semantic similarity. Sometimes users need an exact match, and that's where keyword search becomes important.

Why Vector Search Alone Isn't Enough

Vector search is good at matching meaning, even when the wording differs. For example, "How do I get my money back?" can retrieve a refund policy without the query containing the word "refund."

But vector search is less reliable for exact terms like error codes (ERR_4031), product IDs, function names, and rare acronyms. Someone searching for ERR_4031 likely wants documentation containing that exact error code, not information about similar errors.

Keyword search using BM25 handles these cases better. It ranks documents based on how frequently query terms appear, while giving more weight to rare terms.

Hybrid search combines both approaches. It runs vector search and BM25 in parallel, merges their results, and uses a reranker to select the most relevant chunks.

Each search retrieves a relatively large set of candidates. Fusion combines them into a single ranked list, and the reranker selects the best chunks to include in the model's prompt.

Fusion Methods

Vector search and BM25 produce scores on different scales. A cosine similarity of 0.82 and a BM25 score of 14.3 aren't directly comparable, so we can't simply add them together.

There are two common ways to combine the results.

Reciprocal Rank Fusion (RRF) uses ranking positions instead of raw scores:

For example, a chunk ranked first in vector search and third in BM25 receives a score of 1/61 + 1/63. This rewards chunks that rank highly in multiple result lists.

RRF doesn't require score normalization or learned weights, making it a reliable starting point.

Weighted score fusion normalizes scores to a common range and combines them using configurable weights. For example:

With proper normalization and tuning, weighted fusion can outperform RRF on a specific dataset. However, the weights may need adjustment when the document collection or embedding model changes.

Rerankers

The embedding model used for retrieval is typically a bi-encoder. It encodes the query and each document chunk independently, allowing document embeddings to be computed and stored ahead of time. This makes retrieval fast, but the model doesn't directly compare the query and document while generating their embeddings.

A cross-encoder reranker works differently. It processes the query and each candidate chunk together, producing a relevance score based on their relationship.

For example, consider the question "Can I cancel after 30 days?" A cross-encoder can distinguish a cancellation policy from an article about a 30-day free trial, even though both contain similar terms.

Cross-encoders generally provide better relevance ranking, but they're more expensive because each query-chunk pair must be evaluated at request time.

That's why reranking happens after retrieval. Instead of evaluating millions of chunks, we retrieve 50 to 100 candidates using faster search methods, rerank them, and select the top 5 to 10 for the prompt.

There are several options:

These latency estimates are illustrative. Actual performance depends on the model, candidate count, document length, hardware, and batching.

A reranker can improve retrieval quality without changing the embedding model or index. Measure the gain on your evaluation set.

The main trade-off is candidate count. More candidates give the reranker a better chance of finding relevant information, but increase computation and latency. Start with 50 to 100 candidates and tune the number based on retrieval quality and latency measurements.

So far, we've assumed the user's question already works well as a search query.

Users phrase questions naturally, but those questions don't always work well as search queries. They may depend on earlier messages, combine multiple questions, or use terminology that differs from the source documents.

Query rewriting addresses this by turning the user's question into a clearer, standalone search query. For example, in a conversation, "What about for contractors?" makes little sense on its own. Using the conversation history, a small, fast model can rewrite it as "What is the parental leave policy for contractors?"

Query decomposition breaks complex questions into smaller, focused queries. For example, "How does Q3 churn compare to Q2, and what drove the change?" can be split into separate searches for Q2 churn, Q3 churn, and the factors behind the change. These searches can run in parallel, with their results combined before generating the answer.

HyDE (Hypothetical Document Embeddings) takes a different approach. The model generates a hypothetical answer to the question, then embeds that answer instead of the original query. Because the generated text resembles the documents being searched, its embedding may be closer to relevant content. However, if the hypothetical answer contains incorrect assumptions, it can steer retrieval toward the wrong documents.

These techniques introduce additional model calls and latency, often a few hundred milliseconds depending on the model and deployment. Query rewriting is particularly useful for multi-turn conversations, while decomposition and HyDE are best added when evaluations show they improve retrieval quality enough to justify the extra cost and latency.

The RAG pipeline we've covered so far retrieves information once before generating an answer. This works well for straightforward questions, but it can perform unnecessary searches for messages like "Thanks, that helps" and struggle with questions that require multiple rounds of retrieval.

Agentic RAG gives the model access to retrieval as a tool. Instead of always searching upfront, the model decides whether a search is needed, what to search for, and whether to search again based on the results.

Consider the question, "Which enterprise customers are affected by the authentication outage?"

The model might first search the incident report to identify the affected region, then query the customer directory to find enterprise customers in that region. The second search depends on what the first search reveals, making this a good use case for agentic retrieval.

The model can also choose between different tools, such as document search for incident reports, SQL queries for structured customer data, or web search for external information.

The trade-off is higher and less predictable latency and cost. Each additional round of retrieval may require another model call, and multi-step execution makes debugging and tracing more complex.

A practical approach is to use single-shot retrieval for straightforward questions and allow additional searches when the available evidence is insufficient. Set clear limits on tool calls, retrieval steps, and token usage to prevent unnecessary loops and control costs.

A RAG system can only return information that is in its index. If a pricing page changed yesterday but the index still contains last month's version, the assistant may return an outdated price and even cite the old document as evidence.

Incremental Indexing

Instead of re-indexing the entire document collection, process only what has changed. How you detect changes depends on the source:

The queue absorbs bursts of changes, such as a bulk document import, while ingestion workers process updates independently.

Workers can compare content hashes to identify unchanged chunks and avoid generating embeddings again. This matters because embedding large volumes of text can be one of the more expensive parts of ingestion.

Workers should also handle duplicate and out-of-order events, ensuring an older document version doesn't overwrite a newer one.

Deletes

Deleted documents need special attention. If a delete event is missed, the document may remain searchable and appear in generated answers long after it was removed from the source.

Two practices help prevent this:

Some vector indexes, including certain HNSW implementations, mark deleted vectors rather than removing them immediately. These vectors stop appearing in search results but may continue consuming storage or memory until compaction or an index rebuild.

Different sources also have different freshness requirements. A support knowledge base might need updates within minutes, while an archive could be refreshed weekly.

Define a freshness target for each source and monitor index lag, the time between a source document changing and that change being reflected in search results.

Keeping content current is one part of the problem. Keeping access permissions current is equally important, and mistakes there can expose sensitive information.

In enterprise applications, different users have access to different documents. An engineer shouldn't be able to retrieve confidential HR salary data simply by asking the assistant about it. Access control must be enforced by the retrieval system, not left to the language model.

During ingestion, store each document's access control list (ACL) as metadata alongside its chunks. This might include the users or groups allowed to read the document. At query time, resolve the user's permissions and use them to filter search results.

The important decision is when to apply that filter.

Filtering during retrieval applies the user's permissions as part of the search. Only authorized chunks are returned, so downstream components never receive restricted content.

Filtering after retrieval searches the entire index and removes unauthorized chunks afterward. If filtering happens immediately and correctly, restricted content won't reach the model. However, delaying the permission check until after other processing stages creates several risks:

Post-filtering can also hurt retrieval quality. If most of the top 50 results are restricted, only a few chunks remain, even when relevant, authorized documents exist further down the ranking.

The safer approach is to enforce permissions during retrieval and scope caches by user, tenant, or permission set. Prompt instructions like "Don't reveal confidential information" are not a substitute for access control.

Permission changes also need to propagate quickly. They can use the same change pipeline as document updates, but usually require only metadata changes, not re-embedding.

For systems where even a short delay in permission updates is unacceptable, recheck the final retrieved chunks against an authoritative permission source before passing them to the model. This adds latency but helps prevent access to documents whose permissions have recently changed.

RAG failures are often difficult to detect. The model can produce a fluent, confident answer even when the retrieved information is incomplete, outdated, or incorrect, and error rates and latency look normal.

Here are common failure modes and how to address them:

When debugging an incorrect answer, start by checking whether the relevant information was retrieved.

If the correct chunk is missing, investigate ingestion, chunking, indexing, and retrieval. If the correct chunk is present but the answer is still wrong, examine how the context was assembled, the prompt instructions, and the model's response.

This is why retrieval and answer quality should be evaluated separately. Use metrics like recall@k to measure whether relevant chunks appear in the top results and mean reciprocal rank (MRR) to measure how highly the first relevant result ranks.

A labeled set of representative questions and expected sources makes it easier to identify where failures occur and measure whether changes actually improve the system.

Modern language models can process hundreds of thousands of tokens, with some supporting context windows of over a million tokens. So why not put all the documents directly into the prompt and skip retrieval entirely?

Both approaches have advantages and trade-offs:

Cost is often a major factor. Suppose input tokens cost $3 per million. A 500,000-token prompt costs $1.50 per query, while a 5,000-token RAG prompt costs just $0.015, excluding retrieval costs and output tokens.

Prompt caching can reduce the cost of long-context requests when the same content is reused across queries. However, it doesn't eliminate context window limits or the challenges of processing large inputs.

Long context works well for small, relatively fixed collections, such as a contract, a small codebase, or documents uploaded by a user for a single session.

RAG is better suited for large, frequently changing, or permission-controlled document collections where including everything in the prompt would be impractical.

You can also combine both approaches. Retrieval identifies the most relevant documents, while a large context window allows the model to process complete documents or larger sections instead of relying on small, isolated chunks.

RAG allows language models to answer questions using information retrieved at request time, rather than relying solely on their training data. Here are the key design decisions at each stage:

In a system design interview, start with the two pipelines: how documents are ingested and indexed, and how a user's question becomes an answer. Then focus on the requirements that make the system challenging, such as keeping rapidly changing information up to date, enforcing document-level permissions, or improving retrieval accuracy when incorrect answers have serious consequences.

Build the simplest system that meets the application's requirements for answer quality, latency, cost, freshness, and security.