In a traditional backend service, you can usually verify correctness with automated tests. If a function is expected to return 42 and it returns 42, the test passes. GenAI systems are harder to evaluate because there often isn't a single correct answer. Ask the same question twice, and the model might produce two different but equally valid responses. Even a small change to a system prompt can improve one type of response while unexpectedly making others worse.

This makes evaluation essential to building reliable GenAI systems. Without it, you can't confidently compare models, determine whether a retrieval change improved answer quality, or detect gradual quality degradation in production.

In this chapter, we'll cover how to build evaluation datasets, grade open-ended responses, evaluate RAG systems and agents, and use evaluations as regression tests. We'll also look at how to monitor quality in production and investigate cases where offline evaluation results don't match real-world performance.

Several characteristics of LLM applications make them harder to evaluate than traditional software.

For these reasons, evaluation can't be a one-time activity before launch. It needs to be a continuous process that helps you measure improvements, catch regressions, and identify problems in production.

In production GenAI systems, evaluation is typically a continuous cycle with two parts. Offline evaluation runs before deployment to determine whether a change improves quality or introduces regressions. Online evaluation monitors real user interactions to check whether the system performs as expected in production. Failures discovered in production are added to the offline evaluation dataset, helping the test suite improve over time.

Feeding production failures back into the evaluation dataset keeps this loop useful. Without that step, your test suite gradually becomes outdated, measuring problems that mattered at launch rather than those users encounter today.

The rest of this chapter covers each stage of this process, starting with the evaluation dataset that forms its foundation.

A golden dataset is a curated set of test inputs with expected outputs or grading criteria. Every proposed change is evaluated against this dataset, and its scores are compared with the current production version. The reliability of your evaluation depends heavily on how well this dataset represents real-world usage.

What a Test Case Contains

Each test case typically includes:

Tags are especially important because aggregate scores can hide regressions. For example, an overall accuracy of 87% might look healthy even if accuracy on billing questions dropped from 90% to 70%. Breaking down scores by category helps catch these issues.

Where the Examples Come From

A reliable evaluation dataset typically combines four sources.

Production traces become a valuable source once the system has real traffic. Instead of relying entirely on random sampling, sample across different categories so rare but important requests aren't overlooked. Prioritize conversations involving negative feedback, repeated attempts, or support escalations. Human reviewers should then verify the expected answers or grading criteria.

Synthetic generation is particularly useful before launch. For example, you can give an LLM a document and ask it to generate questions that the document can answer. This also provides a known source document for evaluating retrieval.

The limitation is that synthetic questions often resemble the wording of the source document, making retrieval easier than it would be with real user queries. Review a sample manually, introduce realistic variations, and gradually add real production examples as traffic grows.

How Big Should It Be?

A well-labeled dataset of 100 to 200 examples covering the main use cases is a reasonable starting point for catching major regressions. As the system matures, the dataset may grow to several hundred or a few thousand examples, with new cases added from production failures.

Dataset size also affects how confidently you can interpret score changes. With 100 examples, an accuracy increase from 82% to 85% represents just three additional passing cases. That difference may reflect sampling variation or run-to-run variability rather than a meaningful improvement.

To detect smaller improvements reliably, you generally need more representative examples, repeated evaluations to measure output variability, or both. Comparing the same test cases across versions also helps reduce noise.

Keeping the Dataset Reliable

Two practices help maintain evaluation quality over time.

First, version the dataset like code. Every evaluation result should reference a specific dataset version, so you can distinguish actual system improvements from changes in the test cases or grading criteria.

Second, maintain a held-out test set that engineers don't use while tuning prompts or models. When developers repeatedly optimize against the same failing examples, the system can overfit to those cases. Scores may improve without corresponding gains in production.

With a reliable dataset in place, the next step is deciding how to score the model's responses.

Evaluation methods vary in cost, speed, and flexibility. Simple checks are fast and reliable but limited in what they can measure. LLM-based and human evaluations can assess more subjective qualities, but they're slower and more expensive.

Use the simplest method that can reliably measure each quality criterion, and reserve more expensive methods for cases that require them.

Exact Match and Code-Based Checks

When an output has a single correct answer, you can compare it directly against the expected result. This works well for classification labels, extracted fields, yes/no decisions, and multiple-choice answers.

Even for open-ended responses, many properties can be validated with code:

These checks are deterministic, inexpensive, and easy to automate, although executing code or tests may take additional time.

They should be your first layer of evaluation. Before using an LLM to grade something, check whether you can validate it reliably with code.

LLM-as-Judge

Some qualities, such as factual correctness, relevance, tone, and whether an answer is supported by its sources, require understanding natural language. A common approach is to use a capable LLM to evaluate another model's output.

There are two main approaches:

LLM judges are useful, but they have biases that can affect evaluation results.

Specific grading criteria make a judge more reliable.

For example, asking "Rate this answer from 1 to 10" leaves too much room for interpretation. Instead, ask questions such as "Does the answer correctly state that the refund window is 30 days?" and "Does it mention that digital purchases are excluded?"

Breaking quality into specific, verifiable criteria produces more consistent and actionable results. You can also ask the judge to provide a brief justification with supporting evidence, making incorrect grades easier to investigate.

Calibrating the Judge

An LLM judge can make mistakes, so its scores shouldn't be trusted without validation.

Start by having domain experts label a representative set of a few hundred examples. Run the judge on those same examples and compare its decisions with the human labels.

Percentage agreement is a useful starting point, but it can be misleading when most examples belong to the same category. A chance-corrected metric such as Cohen's kappa provides additional insight, although it also needs careful interpretation with imbalanced labels.

Ideally, the judge should achieve agreement with expert labels comparable to the agreement between human reviewers. However, acceptable performance depends on the application and the cost of incorrect judgments.

If agreement is too low, refine the rubric, add examples of borderline cases, or use a stronger judge model. Pay particular attention to cases where the judge incorrectly passes a bad response.

Also, pin the judge to a specific model version. Otherwise, a provider update could change evaluation scores even when the application being evaluated hasn't changed.

Human Review

Human review provides the reference judgments used to validate other evaluation methods. It's expensive and difficult to scale, so teams typically reserve it for tasks where human expertise adds the most value: labeling golden datasets, calibrating LLM judges, reviewing high-stakes outputs, and auditing samples of production traffic.

Here's how the methods compare:

Costs and latency vary with the complexity of the evaluation, model choice, and review process.

These methods form the foundation of GenAI evaluation. However, RAG systems and agents have additional components that can be evaluated separately, making it easier to identify exactly where failures occur.

A RAG system can produce an incorrect answer for two main reasons: retrieval failed to find the right information, or the model failed to use the retrieved information correctly. A single end-to-end score can't distinguish between these failures, so we need to evaluate retrieval and generation separately.

Retrieval metrics measure whether the system finds relevant information. Faithfulness checks whether the generated answer is supported by that information, while relevance and correctness evaluate the final response.

Retrieval Metrics

Retrieval evaluation requires knowing which documents or chunks are relevant to each query. These labels are typically stored in the golden dataset.

Once the relevance labels are available, these metrics can be calculated without additional LLM calls. This makes them useful for comparing chunking strategies, embedding models, search configurations, and reranking approaches.

Generation Metrics

After evaluating retrieval, we need to measure how well the model uses the retrieved information.

Correctness evaluation usually requires reference answers, while faithfulness and relevance can be evaluated using the query, generated response, and retrieved context. This makes them useful for evaluating sampled production traffic, even when labeled answers aren't available.

Reading the Results Together

Evaluating retrieval and generation separately makes it much easier to identify the cause of poor answers.

These patterns are diagnostic clues rather than definitive explanations. For example, an answer can have high recall and faithfulness but still be incomplete or fail to address the user's intent.

Before making changes, identify whether the failure comes from retrieval, generation, or the underlying data. Otherwise, you might spend time tuning prompts when the real problem is poor chunking or missing documents.

The same principle applies to AI agents, where failures can occur across multiple reasoning and tool-execution steps.

Unlike a RAG system that generates an answer, an AI agent takes a series of actions to complete a task. This means we need to evaluate both the final outcome and the steps the agent took to get there.

Task Success Rate

The primary metric is task success rate: the percentage of tasks the agent completes successfully.

The most reliable way to measure success is to verify the final state of the environment rather than relying on what the agent claims it accomplished.

For example, if you're evaluating a coding agent, run the test suite against its changes. If you're evaluating a support agent that processes refunds, verify that the correct refund was created in the database with the expected amount. Agents can report success even when they haven't completed the task, so their final response alone isn't reliable evidence.

To make these evaluations repeatable, use an isolated environment that can be reset before each run. This could be a fresh repository checkout, a seeded test database, or mocked external APIs with predefined responses.

Reliability Across Runs

Agent behavior can vary between runs, even for identical inputs. A task that succeeds once might fail the next time, so evaluating each task only once isn't enough.

Two metrics help measure reliability:

For example, suppose an agent has a 70% success rate per attempt. Assuming independent attempts:

The agent has a high chance of succeeding at least once in three attempts, but a much lower chance of succeeding every time.

For production agents, consistency is often more important than the ability to succeed after multiple attempts. Repeated evaluations help reveal this difference.

Trajectory Evaluation

Task success tells you whether the agent reached the expected outcome. Trajectory evaluation examines how it got there.

Useful questions include:

Some checks can be automated with code. For example, you can verify that a delete operation was never called or that approval was obtained before processing a refund.

More subjective properties, such as whether the agent followed a reasonable plan, may require an LLM judge to evaluate the execution trace.

Avoid enforcing an exact sequence of tool calls unless the workflow requires it. Different action sequences can be equally valid, so focus on correctness, safety, and efficiency rather than matching a predefined path.

Cost and Efficiency

Task success alone doesn't tell you whether an agent is practical to run in production.

Track the number of steps, tokens consumed, execution cost, and total time required to complete each task. Measure these alongside success rate so you can compare both quality and efficiency.

For example, an agent that succeeds 85% of the time at $0.40 per attempt may be a better choice than one that succeeds 88% of the time at $3.00 per attempt, depending on the application's reliability requirements.

These metrics also reveal regressions that success rates might miss, such as a prompt change that causes the agent to make twice as many tool calls to complete the same task.

Once we can evaluate RAG systems and agents reliably, the next step is integrating these evaluations into the development process so every change is tested before reaching production.

Any change that could affect output quality should trigger an evaluation before reaching users. This includes changes that don't involve application code:

Running the entire evaluation suite with LLM judges on every pull request can be slow and expensive. A practical approach is to divide evaluations into two tiers.

Tier 1 runs on every pull request and should finish within a few minutes. It includes deterministic checks and a small set of critical test cases, enough to catch issues like broken prompt templates, invalid outputs, or incorrect tool arguments.

Tier 2 runs the complete evaluation suite before a release, or nightly for continuous monitoring. It evaluates all categories, uses LLM judges where needed, and compares the results against the current production version.

Setting Release Gates

A release gate determines whether a candidate version is safe to deploy. Since LLM outputs can vary between runs, these gates need to account for statistical noise.

A few practical guidelines:

You also need to account for evaluation costs. Suppose your dataset contains 2,000 examples, with three runs per example and two LLM judge calls per generated response.

That requires:

That's 18,000 model calls for a single full evaluation. Running it nightly may be reasonable, but triggering it on every commit can quickly increase costs.

Offline evaluations help determine whether a change is ready for deployment. However, they can't fully predict how the system will perform with real users. That's where online evaluation comes in.

Once a GenAI system is in production, you need to measure quality using real user behavior and evaluations of sampled interactions. This helps detect problems that offline tests may miss.

Explicit Feedback

Thumbs-up and thumbs-down buttons provide direct user feedback, but usually only a small share of users submit it. Those who do may skew toward negative experiences.

This creates selection bias, making feedback scores unreliable as a measure of overall quality. Instead, use explicit feedback primarily to identify successful interactions and failures worth investigating.

Implicit Signals

User behavior provides additional signals about response quality, often without requiring users to submit feedback.

These signals are indirect and can have multiple explanations, so they should be interpreted alongside other metrics.

Choose signals based on the application's purpose. For example, a coding assistant might prioritize suggestion acceptance rate, while a customer support assistant might focus on resolution rate, escalation rate, and reopened tickets.

Sampled Grading on Live Traffic

You can also use LLM judges to evaluate a small sample of production interactions.

Metrics such as faithfulness and answer relevance don't require reference answers, making them suitable for evaluating real responses. For example, you might sample 1% to 5% of production traces and evaluate them asynchronously to limit cost and avoid adding latency to user requests.

Continuous sampling helps detect gradual quality degradation, such as declining faithfulness caused by outdated documents or changes in user query patterns.

A/B Tests and Canary Releases

Offline evaluations help identify promising changes, but A/B testing measures whether those changes improve real user outcomes.

Randomize traffic by user rather than by individual request, so each user experiences a consistent version. Define the primary success metric and guardrail metrics before starting the experiment.

For example, an A/B test might aim to improve customer support resolution rates while ensuring that customer satisfaction and escalation rates don't get worse.

For higher-risk changes, such as switching models, use a canary release. Start by routing 1% to 5% of traffic to the new version and gradually increase exposure if quality, latency, and error rates remain acceptable.

Another option is shadow testing, where both versions process real requests, but users only receive responses from the existing production version. This lets you compare outputs on real inputs without affecting the user experience, provided the shadow version cannot perform actions that modify production state.

Even with both offline and online evaluation in place, the results won't always agree. A change might improve offline scores while making the real user experience worse, which is why understanding the difference between these signals matters.

Suppose a new prompt improves offline correctness by 5 percentage points, but an A/B test shows no improvement in resolution rate, or even a slight decline. This can happen, and it often points to a gap between what your offline evaluations measure and what users actually experience.

The key distinction is that offline metrics are proxies for quality, while online metrics measure how users respond to the system. Neither is perfect, but offline evaluations should help predict meaningful improvements in production.

When results disagree, investigate the difference rather than assuming one metric is correct. Review examples where the new version performs well offline but poorly in production, identify what the evaluation missed, and update the dataset or grading criteria accordingly.

Over time, this feedback loop makes offline evaluations better predictors of real-world performance.

Keep in mind that online metrics also contain noise. Low traffic, seasonality, changes in user behavior, and differences in traffic composition can affect results even when the system hasn't changed. Before drawing conclusions, check the experiment's statistical uncertainty and whether the observed difference is large enough to matter in practice.

In a system design interview, evaluation should be tailored to the system you're designing rather than presented as a generic list of metrics.

A complete answer covers four areas: how you build the golden dataset, how you measure output quality, what criteria a change must meet before release, and how you monitor quality in production.

For example, when designing a RAG-based customer support assistant, you could explain it like this:

"I'd start with around 300 representative support questions covering the main categories, each with a verified reference answer and the relevant knowledge base articles. I'd measure retrieval using recall@5 and generation using faithfulness and correctness, with an LLM judge calibrated against human-labeled examples.

Every prompt, model, or retrieval change would run against this dataset, with release gates for critical failures and meaningful regressions in individual categories.

In production, I'd monitor resolution rate, escalation rate, and reopened tickets, while running faithfulness checks on a sample of live conversations. I'd also review production failures and add representative cases to the golden dataset so our evaluations stay relevant as user behavior changes."

If evaluation is the main focus of the interview, such as when designing a medical summarization system or an agent that performs real-world actions, go deeper into judge calibration, statistical confidence, category-specific release gates, and how to investigate disagreements between offline and online results.

Start with a representative dataset, use the simplest grading method that gives reliable scores, test every change that affects quality before release, and add production failures to your evaluation datasets as you find them.