Language models do not automatically know your product catalog, incident reports, customer records, or yesterday's policy change. A model can only answer from information available in its weights, in the prompt, or through tools connected to the application. Retrieval-Augmented Generation (RAG) is the most common pattern for giving a model the right information at the moment it needs it.

RAG combines information retrieval with language generation. Instead of relying on whatever the model absorbed during training, the application first retrieves relevant evidence from external sources such as documents, tickets, databases, code repositories, or knowledge bases.

That retrieved evidence is then included in the model input. The model still generates the answer, but it is asked to work from supplied context rather than memory. When retrieval is good and the prompt is clear, this produces answers that are easier to verify and update.

In this chapter, we will cover how RAG works, why it became a standard AI engineering pattern, when it is a better fit than fine-tuning or long-context prompting, and where basic RAG is not enough.

Consider a customer support chatbot for an e-commerce company. It needs to answer questions like:

The first question might be answerable if the return policy happened to appear in the model's training data. But the other three require information that is either private (order status), constantly changing (promotions), or too specific to appear in any public dataset.

This is the knowledge freshness problem. It shows up in three practical ways.

Temporal freshness

The model's training data has a cutoff date. Information created after that date is not in the model's weights. A model trained in March 2025 will not know about a policy changed in April 2025 unless the application provides that information. It cannot reliably answer about new product launches, updated regulations, or recent security vulnerabilities from memory alone.

Organizational freshness

Your internal knowledge, company policies, engineering docs, customer data, and meeting notes were never in the training data to begin with. Unless your application provides that information, the model has no reliable access to your organization's context.

Domain freshness

Even within its training data, the model's knowledge is uneven. It knows more about common public topics than about niche domains. If your application deals with specialized medical procedures, obscure legal regulations, or proprietary engineering standards, the model's coverage may be thin.

Missing knowledge does not automatically produce silence. A model may still produce a fluent answer when the evidence is absent, because language models are trained to continue text. Without retrieval, abstention rules, and verification, that gap can turn into confident, unsupported text.

There are three main approaches to this problem, and the trade-offs between them shape most of the decisions that follow.

Approach 1: Fine-Tuning

Fine-tuning takes a pre-trained model and continues training it on your specific data. You give it examples from your documentation, Q&A pairs, domain language, or task workflow, and the model updates its weights.

This can sound like the obvious solution: if the model does not know your data, train it on your data. In practice, fine-tuning solves a narrower problem.

Fine-tuning is usually best for teaching a model how to respond: tone, format, tool-use patterns, domain-specific phrasing, or task behavior. It is usually a poor primary mechanism for teaching a model what facts are true today.

When you fine-tune a model on your documentation, the information gets baked into the model's weights. It becomes part of the model itself. That means every time your documentation changes, you may need another training and deployment cycle, whether the change is a new product feature, an updated policy, or a fixed typo in your API docs.

Each fine-tuning run requires data preparation, evaluation, deployment, and rollback planning. For a company that updates its docs weekly or daily, that is the wrong operational loop.

There is also a subtler risk: a poorly designed fine-tune can degrade general behavior or overfit to stale examples. Modern instruction-tuning methods reduce this risk, but they do not eliminate the maintenance problem.

Approach 2: Retrieval-Augmented Generation (RAG)

RAG takes a different approach. Instead of baking knowledge into the model, you keep the knowledge separate and provide it at query time.

When a user asks a question, you first search your knowledge base for relevant documents. Then you include those documents in the model's prompt alongside the question. The model reads the provided context and generates an answer based on it.

The model itself never changes. What changes is the input you construct for each query.

This separation of concerns is the reason RAG is so useful. The model interprets the question and generates the answer. The retrieval system handles storage, filtering, ranking, access control, and freshness. Update a document, re-index the affected chunks, and the next query can use the new information without retraining the model.

Approach 3: Long Context Windows

Modern language models support large context windows. Many production models can handle very large prompts. So why not put the whole knowledge base into the prompt?

For small knowledge bases, this can work well. If your entire documentation fits comfortably in the prompt, long context is simpler than building a RAG pipeline. No chunking, no embeddings, no vector database. You include the relevant documents and ask your question.

But this approach hits practical limits. Cost and latency grow with prompt size. Long prompts also make access control, citations, and source selection harder. Recent long-context models are better than earlier systems at using information buried deep in a prompt, but retrieval is usually cheaper and easier to control for large or frequently changing corpora.

Comparing the Three Approaches

Here is how the three approaches stack up across the dimensions that matter most in practice.

For many applications where a model needs to answer from private or changing data, RAG is the right starting point. It handles large corpora, supports targeted updates, keeps per-query cost bounded, and gives you source-level observability.

These approaches are not mutually exclusive. Many production systems combine them. You might fine-tune for response style and use RAG for factual retrieval. Or use long context for a small set of important documents and RAG for the rest.

RAG has two phases: an offline phase where you prepare your knowledge base, and an online phase where you answer questions.

The Offline Phase: Indexing Your Documents

Before you can retrieve anything, you need to turn your documents into something searchable. This happens once upfront, with updates whenever your content changes.

You load your documents, split them into chunks, convert each chunk into a vector embedding, and store everything in a vector database. A chunk is a smaller piece of content that can be retrieved on its own. Chunking strategy has a large impact on retrieval quality, but we will keep things simple for now and build more capable pipelines later.

The Online Phase: Answering Questions

When a user asks a question, here is what happens step by step.

The pattern is simple, but the quality comes from engineering discipline. The model does not need your domain knowledge in its weights. It needs the right evidence, formatted clearly, with instructions about what to do when that evidence is insufficient.

The modern RAG formulation was introduced in a 2020 paper by Lewis et al. at Facebook AI Research. Since then, the pattern has become a standard architecture for knowledge-grounded language-model applications because it matches the operational shape of real software systems.

Separation of concerns

RAG separates knowledge management from answer generation. This means you can update knowledge without touching the model, and upgrade the model without rebuilding your knowledge base.

Fast updates

When a document changes, you re-embed the affected chunks and update the vector store. The next query can use the new information. Compare this to fine-tuning, where an update usually means another training and deployment cycle.

Cost efficiency

You usually send only the retrieved chunks to the generation model, not your entire knowledge base. A query that retrieves 5 chunks of 200 tokens each has a bounded prompt size, even if the knowledge base contains thousands of documents.

Transparency

You can inspect which documents were retrieved and passed to the model. This enables citations, source attribution, and auditing, which are much harder when knowledge is encoded only in model weights.

No retraining side effects

The model itself is not modified. You are augmenting it with context. A RAG-powered customer support bot can still use the base model's general language ability because the underlying model has not been altered.

Works with many models

RAG is mostly model-agnostic. You can often swap the generation model or provider without rebuilding the knowledge pipeline, as long as the new model can follow the same grounding and citation instructions. The retrieval layer can stay the same.

RAG is useful, but it is not a universal answer. There are cases where it adds complexity without solving the real problem.

When you need to change the model's behavior, not its knowledge

If you want the model to respond in a specific tone, follow a strict format, or reason in a domain-specific way, fine-tuning is more appropriate. RAG gives the model information. Fine-tuning changes how it processes information.

When your knowledge base is very small

If your entire documentation fits comfortably in the prompt, long context is simpler. You skip the complexity of building a retrieval pipeline and include the relevant documents directly. The cost difference may be small at this scale.

When retrieval quality is fundamentally unreliable

Some corpora contain documents that are nearly identical except for details that matter. Legal clauses, medical protocols, API versions, and code snippets often differ by one word, number, or condition. Basic semantic search can retrieve a plausible but wrong neighbor. These systems need keyword search, metadata filters, reranking, version constraints, and sometimes symbolic retrieval in addition to embeddings.

When real-time data is required

RAG usually works against a pre-indexed knowledge base. If the answer depends on live state (current stock prices, sensor readings, order status, inventory, account balance), call the source of truth directly. You can combine tool calls with RAG, but do not treat yesterday's index as real-time data.

When the question requires complex reasoning across many documents

Standard RAG retrieves a handful of chunks and asks the model to synthesize an answer. If the answer requires combining information from dozens of documents with multi-step reasoning, basic RAG may not be enough. You may need patterns such as multi-hop retrieval, graph-based retrieval, or a more structured workflow.

Let's build a small working RAG system. This implementation is deliberately minimal: no framework, no extra abstractions, only the core pattern.

We will use ChromaDB as our vector store, OpenAI's text-embedding-3-small for embeddings, and a configurable chat model for generation.

Let's walk through what this code does.

First, we create a ChromaDB collection and add 10 documents. Each document gets embedded and stored alongside its text. In a real system, these would be chunks from your actual documentation, not handwritten sentences.

The ask function implements the core RAG loop. It embeds the user's question using the same embedding model, queries ChromaDB for the 3 most similar documents, assembles them into a context string, and passes everything to the generation model with instructions to only use the provided context.

Notice the last query: "What programming languages does Acme support?" None of the retrieved documents mention programming languages, so the prompt instructs the model to abstain. RAG does not automatically make a model honest; the retrieval boundary, prompt, and evaluation loop make abstention more likely and easier to test.

This small system captures the core RAG pattern, but it cuts many corners that production systems cannot afford to cut.

No document loading or parsing

We used 10 handwritten sentences. Real systems ingest PDFs, Markdown files, HTML pages, and database records. Each source type needs its own parsing logic.

No chunking

Our "documents" are single sentences. Real documents are pages or chapters that need careful splitting to balance context preservation with retrieval precision. Chunking strategy has a major effect on retrieval quality.

No metadata

We do not track where each chunk came from, what page it was on, or when it was last updated. Without metadata, you cannot provide citations or filter by source.

No error handling

API calls fail. Embeddings take time. Vector stores can be unavailable. Production pipelines need retry logic, timeouts, and graceful fallback behavior.

No query preprocessing

Users ask vague, misspelled, or multi-part questions. Production systems often rewrite or expand queries before retrieval to improve results.

No evaluation

We have no way to measure whether the system is giving good answers. Without evaluation, you cannot tell whether a change improves or degrades answer quality.

These gaps are the difference between a demo and a system users can rely on. The next step is turning this baseline into a production RAG pipeline.

References