A GenAI system can fail without throwing an error. For example, the retriever might return the wrong document and the model might generate a confident but incorrect answer, while the API still responds with HTTP 200. The dashboards look healthy even though users are getting bad answers.

Monitoring a GenAI system therefore needs more than availability checks. Traditional monitoring tells you whether a service is available and performing well. For an LLM application, you also need to know whether the answers are accurate, how much each request costs, and which changes affect response quality.

LLMOps addresses these challenges. It covers tracing requests, versioning prompts and models, deploying changes safely, monitoring quality and cost in production, using production data to improve the system, and handling failures when they occur.

In this chapter, we'll explore these practices and how they work together to help you run reliable GenAI systems in production.

Logs, metrics, and traces are still essential in GenAI systems. You still need to monitor error rates, p99 latency, and resource utilization. But LLM applications introduce three challenges that traditional monitoring doesn't fully address.

Behavior can change without code changes. In a traditional service, behavior typically changes when new code is deployed. In a GenAI system, a small prompt edit, a model upgrade, changes to the retrieval index, or even a provider-side model update can affect responses. These changes may happen without a code deployment, making regressions harder to trace.

Outputs are non-deterministic and difficult to evaluate. The same input can produce different responses, and a successful HTTP response doesn't mean the answer is correct. Measuring quality requires automated evaluations, user feedback, or human review. These signals are less direct and often less reliable than traditional error metrics.

Cost varies significantly between requests. A simple question might cost a fraction of a cent, while an agent executing 15 tool calls with a large context could cost a dollar or more. You need to track token usage and cost at the request level to understand spending and identify expensive operations.

Here's how these differences affect what you monitor:

To understand why an LLM application produced a particular response, you need visibility into every step of the request. That starts with end-to-end tracing, which we'll cover next.

A trace records the steps a system takes to produce a response. In a GenAI application, it should help you answer questions like: What did the model see? Which documents were retrieved? Which tools did the agent call, and what did they return? How many tokens were consumed? What did the request cost, and where was the time spent?

What a Trace Contains

A trace is organized as a tree of spans. The root span represents the entire request, while child spans capture individual operations such as guardrail checks, retrieval, reranking, model calls, and tool executions.

For example, consider a customer support assistant answering a refund question using retrieval and an order lookup tool:

When a user reports an incorrect answer, the trace gives you a starting point for debugging. If the correct refund policy never appears in the retrieved results, you investigate retrieval. If it was retrieved but removed during reranking, you investigate the reranker. If the model received the correct policy but still gave the wrong answer, you examine the prompt and model response.

Without this visibility, it's difficult to identify where the problem occurred.

What Each Span Should Record

Each span captures information specific to its operation. Here are the most useful attributes for debugging:

Full prompts, outputs, and tool results should only be captured when permitted by your data privacy policy. We'll cover this shortly.

Two attributes deserve special attention.

The finish reason tells you why the model stopped generating. For example, if the model hits its maximum output token limit, it may return incomplete JSON. The API call succeeds, but downstream parsing fails. Tracking finish reasons helps you identify these problems.

The release version identifies the combination of prompts, models, retrieval configuration, and other settings used for a request. This becomes essential when comparing releases, investigating regressions, or rolling back a change.

OpenTelemetry provides semantic conventions for GenAI operations, including attributes such as gen_ai.request.model and gen_ai.usage.input_tokens. Many LLM observability tools support these conventions, allowing you to use the same tracing infrastructure as your other services.

Recording Traces Without Slowing Requests

Trace collection should add minimal overhead to the request path. Instead of sending every span synchronously to a tracing backend, the application should buffer spans locally and export them asynchronously.

If the tracing backend becomes unavailable, requests should continue working. The exporter can retry within a bounded buffer, then drop telemetry if necessary rather than blocking user requests.

A typical collection pipeline looks like this:

Sensitive data should be excluded or redacted before export whenever possible, with additional filtering at the collector.

Aggregated metrics feed dashboards and alerts. Detailed traces go into a searchable store for debugging, while selected traces can be retained longer for building evaluation datasets.

Storage, Sampling, and Privacy

GenAI traces can become expensive to store. A single request containing 10,000 tokens of retrieved context and a long response might generate 50 KB or more of trace data.

At 10 million requests per day, that's approximately 500 GB of trace data per day, before compression.

You can control this with three practices:

Privacy requires just as much attention as storage costs. Traces may contain customer messages, personal information, internal documents, or sensitive tool results. Avoid collecting unnecessary data, redact sensitive fields before storage, restrict access by role, and enforce appropriate retention and deletion policies.

For some enterprise customers, storing only metadata may be the only acceptable option.

Tracing tells you what happened during a request. But when response quality changes after a deployment, you also need to know exactly what changed in the system.

That's where versioning comes in.

When response quality drops, you need to know exactly which prompt, model, and retrieval index produced the earlier answers. Without that information, it's difficult to identify what changed or roll back to a working version.

Every component that affects model behavior should be versioned, and every request trace should record those versions.

What Needs a Version

Prompts. Treat prompt templates like code. Store them in version control or a prompt registry with immutable versions, and reference them by ID at runtime. Avoid editing production prompts in place without version history, as even small changes can introduce regressions.

Models. Pin specific model versions whenever the provider supports it. Some providers offer aliases like latest that can point to newer model snapshots without a deployment on your side. These are convenient during development but risky in production. Treat model upgrades as releases that require evaluation. For self-hosted models, version the model weights, quantization settings, and inference engine configuration together, since all three can affect outputs.

Indexes. A retrieval index depends on the source documents, chunking strategy, and embedding model. Changing the embedding model generally requires re-embedding the documents and updating the query encoder to match. Use versioned index names, such as support-docs-v14, and keep the previous index available until the new one has been validated.

Configuration. Settings like temperature, maximum output tokens, retrieval top-k, reranking thresholds, tool definitions, and guardrail rules can all change system behavior. Keep them in versioned configuration rather than scattered across environment variables.

The Release Manifest

Versioning individual components isn't enough because system behavior depends on how they work together. A prompt that performs well with one model might produce worse results with another.

A practical solution is a release manifest: a versioned configuration that defines the exact combination of components used by the application.

Every request trace records the release ID. If response quality drops, you can compare metrics across releases and identify whether the regression coincides with a particular change.

Rollback also becomes simpler. Instead of reverting individual components, you can route new requests back to the previous release manifest.

Limits of Rollback

Rolling back a prompt or switching to a previous model version can be straightforward, but only if the older versions are still available. Some changes are harder to reverse:

Versioning makes it possible to identify regressions and return to a known working configuration. The next step is reducing the chance of bad changes reaching users in the first place through safe deployment and evaluation practices.

Small changes to prompts, models, or retrieval can affect response quality in unexpected ways. To reduce this risk, roll out changes gradually, validating each stage before expanding traffic.

A typical deployment pipeline looks like this:

Not every change needs every stage. A minor prompt correction might move directly from offline evaluation to a canary, while a major model migration usually benefits from the full pipeline.

Offline Evaluation as the First Gate

Before exposing users to a new release, evaluate it against a fixed test dataset and compare the results with the current production version.

The dataset should cover common requests, difficult edge cases, and previously identified failures. Check individual cases for regressions as well as the average score. A change that improves overall quality but breaks cases that previously worked may not be ready to ship.

Offline evaluations provide a fast way to catch problems, but they can only test scenarios represented in the dataset. Real users introduce much more variation, which is why production validation is still necessary.

Shadow Traffic

Shadow testing sends copies of real production requests to the candidate release while users continue receiving responses from the current version.

You can compare the outputs using automated graders and measure differences in latency, token usage, response format, and refusal rates without exposing users to the new behavior.

Shadow testing is particularly useful for model migrations and retrieval changes, but it has two limitations.

First, running both versions increases inference costs, so teams usually shadow only a sample of requests.

Second, shadow testing becomes more complicated for agents that execute actions. You cannot let a shadow agent send actual emails, issue refunds, or place orders. Its tools must be read-only, mocked, or isolated in a sandbox, which can make the results less representative of production.

Canaries

A canary rollout sends a small percentage of real traffic, typically 1% to 5%, to the new release. If the results remain healthy, you gradually increase that percentage.

Alongside standard error and latency metrics, monitor GenAI-specific signals such as guardrail trigger rates, refusal rates, output length, cost per request, user feedback, and automated quality scores.

The main challenge is collecting enough data to detect meaningful regressions.

For example, only a small percentage of users may provide feedback. A 2% canary on a low-traffic feature might need several days to reveal a statistically meaningful drop in satisfaction.

Operational metrics such as latency, cost, malformed outputs, and refusal rates are usually available much sooner. These are good candidates for automatic rollback rules, while quality metrics may need longer observation periods or additional review.

Canary assignment should also be sticky at the user or conversation level. If the same conversation switches between two model or prompt versions, responses may become inconsistent, making the results harder to compare.

Feature Flags

Feature flags let you deploy a new release without immediately exposing it to everyone.

You can enable it for internal users first, then selected tenants, and eventually a larger percentage of traffic. If something goes wrong, you can disable the release without deploying new code.

In GenAI applications, feature flags can also control individual components. For example, you might use them to switch model versions, enable an expensive reranker, or temporarily disable an agent tool that's causing problems.

Here's how these techniques compare:

These techniques work best together. Offline evaluations catch known problems, shadow testing exposes differences on realistic inputs, canaries validate behavior with real users, and feature flags control exposure.

But even carefully tested releases can develop problems in production, whether from changing user behavior, new data, or provider-side updates. Continuous monitoring is how you detect those problems before they affect too many users.

Monitoring a GenAI system in production involves four areas: operational health, cost, provider reliability, and response quality. The first three can be measured with familiar metrics. Quality is harder because a technically successful request can still produce a bad answer.

Operational Metrics

For streaming responses, latency has two important components. Time to first token (TTFT) determines how quickly users see a response, while output speed (tokens per second) determines how fast the rest of the answer arrives.

Track both at p50 and p95, broken down by model and feature. Also monitor total request latency, especially for RAG pipelines and agents that perform multiple steps before generating a response.

These breakdowns help identify bottlenecks. A large prompt, a slow model, and a delayed tool call can all increase total latency, but each requires a different fix.

Other useful metrics include structured-output parsing failures, responses truncated by token limits, tool-call error rates, and the number of steps per agent run.

Cost Metrics

LLM costs vary significantly between requests, so you need to monitor them at a granular level. The LLM gateway is a good place to track usage because it sees the model selected and the tokens consumed by each call.

Track cost per request, feature, tenant, and agent run. Break down input, output, and cached tokens separately, since they often have different prices.

Cost spikes usually have identifiable causes. A prompt update might double the input tokens. Putting a dynamic timestamp at the beginning of a prompt could reduce cache hits. An agent might get stuck in a loop, or a single tenant might suddenly generate much more traffic.

Breaking down spending by model, feature, tenant, and release helps turn an alert like "cost increased by 40%" into something you can investigate and fix.

Provider Health

When your system relies on hosted models, provider reliability directly affects your application's availability and performance.

Monitor rate-limit responses (HTTP 429), server errors, timeouts, and latency separately for each provider and model.

An increase in 429 responses can indicate that you're approaching a quota or hitting a burst limit. A sudden increase in TTFT from one provider may indicate degraded performance, even before an incident appears on its status page.

These metrics also help you decide when to reduce traffic, adjust concurrency, or switch to a fallback provider.

Quality Signals

Response quality is harder to measure because no single metric captures whether an answer is useful, accurate, and appropriate.

In practice, teams combine several signals, each with different strengths and limitations:

A practical approach is to collect inexpensive proxy metrics for every request, run automated graders on a sample of responses (for example, 1% to 5%), and have humans review a smaller set.

Human review also helps calibrate automated graders. If an LLM judge consistently rates incorrect answers as good, its scores won't tell you much about actual quality.

Drift

Response quality can decline even when you haven't deployed anything.

Users might start asking about a newly released product, a data connector might stop syncing, or a hosted provider might update the model behind an alias.

There are two useful types of drift to monitor.

Input drift happens when the types of requests users send begin to change. You can embed queries, group them into topic clusters, and track whether new or growing clusters have lower evaluation scores or more negative feedback.

Data drift happens when the information available to the system becomes outdated or no longer matches user needs. Monitor document ingestion failures, index freshness, and retrieval quality. A sustained decline in retrieval relevance for a particular topic may indicate that the required content is missing or outdated.

Alert on distribution changes that coincide with declining quality, rather than on every shift in the data.

Choosing What to Alert On

Not every metric deserves an immediate alert. Separate fast, actionable signals that may require an on-call response from slower quality trends that benefit from periodic review.

Use short monitoring windows for operational problems such as provider outages and latency spikes. Quality signals usually need longer windows and enough samples to distinguish real regressions from normal variation.

Alerts should also be tied to user impact. A brief increase in provider errors may not require paging if retries and fallbacks are handling them successfully. A sustained increase in failed requests does.

Monitoring helps you detect problems, but the traces, evaluations, and feedback you collect serve another purpose: they provide real examples of where your system needs improvement.

That brings us to the data flywheel, where production data becomes the foundation for improving future releases.

Production traces show how people actually use your system, including the cases it handles well and those where it struggles.

The data flywheel turns these traces and user feedback into evaluation cases and, when appropriate, training data. Over time, this helps improve response quality while making future regressions easier to detect.

Selecting the Right Traces

Random sampling is useful for understanding overall quality, but it often produces examples the system already handles well.

For improvement, focus on traces that reveal problems: negative user feedback, conversations escalated to human support, low grader scores, guardrail triggers, and requests from new or poorly performing topic clusters.

Prioritizing these cases helps you find more meaningful problems with less manual review.

Triage Before Labeling

Before turning a failed request into an evaluation case, identify what went wrong. Different failures require different fixes.

A reviewer can use the trace to classify the problem into a few common categories:

Tracking these categories over time helps identify where improvements will have the greatest impact.

For example, if 60% of failures come from retrieval, improving the prompt is unlikely to solve the underlying problem. You would get more value from improving indexing, search, or reranking.

Turning Traces Into Eval Cases and Training Data

Once a failure has been reviewed, it can become an evaluation case containing the original input, relevant context, and either a reference answer or a grading rubric.

As these cases accumulate, your evaluation dataset becomes more representative of real user problems, including edge cases that weren't anticipated during development.

Some examples can also support model training. Human-corrected responses can become supervised fine-tuning examples, while preferred and rejected responses to the same input can be used for preference optimization.

However, fine-tuning shouldn't be the default response to every quality problem. Fixing a missing document, improving retrieval, or adjusting a prompt is often faster, cheaper, and easier to validate.

There are two important precautions.

Privacy and consent. Production traces may contain personal or confidential information. Only use customer data for evaluation or training when permitted by applicable agreements and policies. Remove or mask sensitive information, and respect tenant-specific restrictions.

Evaluation contamination. Avoid using the same examples for both training and final evaluation. Otherwise, the model may perform well on familiar cases without improving on new requests. Maintain a separate, held-out evaluation set that is never used for training or tuning.

The data flywheel creates a continuous improvement process: identify failures, understand their causes, add regression tests, and validate fixes before deploying them.

But not every problem can wait for the next release. Some failures require an immediate response, which brings us to incident management for GenAI systems.

Incidents in GenAI systems don't always begin with an outage alert. Sometimes the first sign of trouble is a user reporting an incorrect or incomplete answer.

Let's look at three common scenarios and how the observability and LLMOps practices we've covered help detect, investigate, and resolve them.

A Bad Prompt Release

Suppose a team updates a support assistant's system prompt to make responses shorter. Offline evaluations pass, but two days later, users start reporting that the assistant no longer includes return shipping instructions.

Detect: Negative feedback increases for refund-related questions, and online evaluation scores for answer completeness drop on the new release.

Mitigate: Roll back to the previous release manifest. Because prompts are versioned, the team can restore the previous configuration without a code deployment.

Fix and prevent: Review the failing traces and confirm the cause. In this case, the new instruction to "keep answers under 80 words" caused the model to omit required steps. Add these examples to the evaluation dataset, with a rubric that checks for complete return instructions. Future prompt changes will now be tested against these cases.

A Provider Outage

Suppose the primary model provider starts returning server errors and timeouts for 30% of requests.

Detect: Provider error-rate alerts fire, and request latency increases as retries and timeouts accumulate.

Mitigate: The LLM gateway's circuit breaker detects repeated failures and temporarily stops routing requests to the affected provider. Traffic shifts to a fallback model.

If no suitable fallback is available, the system can degrade gracefully by serving valid cached answers for common questions, temporarily disabling nonessential features, or returning a clear error message instead of leaving users waiting.

Fix and prevent: Make sure fallback models are configured and tested before an incident. Each fallback should have compatible prompts, validated response formats, recent evaluation results, and enough quota to handle additional traffic.

An untested fallback may restore API availability while introducing new quality problems or breaking downstream parsing.

A Model That Changed Behavior

Suppose no new code or configuration was deployed, but structured-output parsing failures triple overnight, and responses become noticeably longer.

Detect: Alerts identify the increase in parsing failures and output length. Breaking down the metrics by release shows that the application configuration hasn't changed.

Investigate: Check whether the provider updated the model behind an alias or made another serving-side change. Compare traces from before and after the regression, using similar inputs to identify differences in output behavior.

Mitigate and prevent: Switch back to the previous model version if it's still available. Treat the updated model as a new release, run offline evaluations, adjust prompts if necessary, and roll it out gradually.

To reduce the risk of similar incidents, pin specific model versions whenever possible and run scheduled evaluations to detect unexpected changes in behavior.

The same process applies across all three scenarios: detect the problem through monitoring, investigate using traces and release history, mitigate through rollback or fallback, and add regression cases to prevent recurrence.

LLMOps questions can come up in a GenAI system design discussion, for example "How would you monitor this system?" or "How would you deploy a new model safely?"

A good approach is to explain how you would operate the system in production. Cover what you capture in request traces, how you version prompts and models, how you evaluate and roll out changes, which metrics you monitor, and how production failures become new evaluation cases.

Focus on the areas most relevant to the system you're designing. For a RAG assistant, emphasize retrieval tracing, index versioning, and groundedness evaluation. For an agent, discuss tool-call tracing, cost per run, and safely testing tools that have side effects. For an LLM gateway or inference platform, focus on per-tenant usage and cost tracking, provider reliability, and failover.

Rather than listing observability tools or metrics, walk through how you would detect a problem, find its cause, ship a fix safely, and use production data to improve the system.

LLMOps builds on traditional operations to address the challenges of GenAI systems: behavior that changes with prompts, models, and data; response quality that's difficult to measure; and costs that vary between requests.

Here are the key practices and decisions:

Record the prompt, model, and index versions in every trace, release changes gradually behind metrics you can roll back on, and turn production failures into evaluation cases.