Three Pillars of Observability

Suppose you run an e-commerce application. One afternoon, customers start complaining that checkout is slow. Some orders take 10 seconds to complete. A few fail entirely.

If the whole application ran on one server, you might log into that machine, look at the CPU, and read through the log file.

But modern systems rarely look like that. A single checkout request might pass through an API gateway, an order service, a payment service, an inventory service, a database, and a cache.

So where do you even begin? Which service is slow? Is it slow for every user or only some of them? And what changed?

To answer these questions, the system needs to tell you what is happening inside it. That is what observability is about.

In this chapter, we will learn what observability means, the three types of data it is built on, which are metrics, logs, and traces, and how they work together when you need to debug a problem in production.

You may have heard the term monitoring, so it helps to separate the two ideas.

Monitoring means watching a set of known signals. You decide in advance what to track, such as CPU usage, error rate, or response time. Then you set alerts when those values cross a threshold.

Monitoring works well for problems you already expect. But in distributed systems, many failures are new. Maybe checkout is slow only for users paying with one specific card type, in one region, after a recent deployment. Nobody wrote an alert for that.

Observability is the ability to understand the internal state of a system by looking at the data it produces. A well-instrumented system lets you ask new questions during an incident, without shipping new code first.

In practice, you need both. Monitoring catches the issues you can predict. Observability helps you debug the ones you cannot.

To make that possible, systems typically produce three kinds of telemetry data: metrics, logs, and traces. These are often called the three pillars of observability.

Each pillar is good at answering a different kind of question.

Metrics tell you whether something is wrong. For example, the error rate for checkout jumped from 0.5% to 8%.

Logs tell you what exactly happened. For example, a specific payment request failed with a connection timeout.

Traces tell you where the time went. For example, a checkout request spent most of its time waiting on the fraud check service.

None of them gives the full picture alone. So let's look at each one, starting with metrics.

A metric is a numeric measurement recorded over time.

Most metric systems support a few basic types:

Here is what these look like in a Prometheus-style format:

Metrics also carry labels. A request counter might have labels for the service name, the endpoint, and the status code. That lets you ask questions such as: how many requests to the checkout endpoint returned a 500 error in the last 10 minutes?

Why Metrics Are Cheap

The main strength of metrics is that they are compact. A metric does not store every request. It stores a number that summarizes many requests.

Whether your service handles 100 requests per second or 100,000, the request counter is still a single number updated over time. That makes metrics cheap to store and fast to query. It is why metrics power most dashboards and alerts.

One important detail is how you summarize latency. Suppose 99 requests take 50 ms and one request takes 5 seconds. The average is about 100 ms, which looks healthy. But 1% of your users are waiting 5 seconds.

That is why teams usually track percentiles such as p95 and p99 instead of relying on averages.

A common starting point is the four golden signals: latency, traffic, errors, and saturation. Together they show how long requests take, how much demand the service is under, how often requests fail, and how close resources are to their limits.

The Limits of Metrics

Metrics also have a clear limitation. Because they aggregate data, they lose individual details. A metric can tell you that the checkout error rate is 8%. It cannot tell you which orders failed or why.

You might think the fix is to add more labels, such as a user ID or an order ID. But every unique combination of label values creates a separate time series.

With 10 million users, adding user ID as a label can turn one metric into 10 million time series. This is called the high cardinality problem, and it can overload a metrics system quickly.

So labels should have a small, bounded set of values, such as service name, region, or status code.

When you need details about individual requests, you need a different kind of data. This is where logs come in.

A log is a timestamped record of something that happened in the system.

A service might write a log when it receives a request, when a payment fails, when a retry happens, or when a background job finishes.

Logs are the most detailed of the three pillars. That level of detail is what you need once you know something is wrong and want to understand the exact cause.

Structured Logging

How you write logs matters a lot. Many applications write plain text lines, such as:

That is easy for a person to read, but it is hard to search across millions of entries. The log system has to guess where each value is.

Structured logging solves this by writing each log as a set of key-value fields, usually in JSON. Instead of one sentence, the log has separate fields for the timestamp, the level, the service, the order ID, the error type, and the retry count.

Now you can run precise queries: show me every error from the payment service in the last hour where the error type was timeout.

This makes logs far easier to search, filter, and aggregate during an incident.

The Limits of Logs

Logs have their own trade-offs.

The biggest one is volume. A busy system can produce terabytes of logs per day, and storing and indexing all of it gets expensive. So teams use log levels such as debug, info, warning, and error, and often turn off debug logs in production.

The second problem is that logs are spread across many machines. Containers come and go, and a log file on a container that no longer exists is gone. That is why production systems ship logs to a central system where you can search them all in one place.

But even with centralized logs, one question remains hard. If a request touched six services, how do you connect the log lines those six services wrote about that one request?

That brings us to the third pillar.

A trace follows a single request as it moves through the system.

Suppose a user clicks the checkout button. The request enters the API gateway, which calls the order service. The order service calls the inventory service and the payment service. The payment service calls a fraud check service before charging the card.

A trace records this entire journey.

Each unit of work inside the trace is called a span. The API gateway handling the request is one span. The order service processing it is another span. Each span records when it started, how long it took, whether it succeeded, and which span called it.

Put together, the spans form a tree that shows how the request moved through the system.

A span usually stores a trace ID, its own span ID, its parent span ID, an operation name, start and end times, a status, and key-value attributes that describe the work.

How Trace Context Is Propagated

For this to work, every service needs to know which trace a request belongs to.

When the request enters the system, the first service generates a unique trace ID. When it calls another service, it passes that trace ID along, usually in an HTTP header.

A common standard for this is the W3C Trace Context specification, which defines a header called traceparent:

The next service reads the header, creates its own span under the same trace ID, and passes the context along to any service it calls.

If even one service in the chain fails to pass the context along, the trace breaks into disconnected pieces. That is one reason most teams rely on tracing libraries instead of writing it themselves.

What Traces Reveal

Tracing tools usually display a trace as a waterfall. Each span is a horizontal bar, and its length shows how long it took.

Suppose the checkout request took 4 seconds in total. In the waterfall, you might see that the order service took only 50 ms of its own time, the inventory call took 100 ms, and the fraud check took 3.5 seconds.

Now you know exactly where to look.

Traces can reveal other problems too. You might notice that three service calls run one after another when they could run in parallel. These patterns are very hard to spot from metrics or logs alone.

The Limits of Traces

Like logs, traces can produce a large amount of data. Recording every span for every request in a high-traffic system is often too expensive. So most systems use sampling.

With head-based sampling, the decision is made when the request starts. For example, the system might keep 1% of all traces. This is simple and cheap, but it may throw away the rare slow or failed request you actually care about.

With tail-based sampling, the system waits until the request finishes and then decides whether to keep it. That way, it can keep every trace with an error or high latency, and only a small sample of normal ones.

The cost is that the tracing infrastructure has to hold spans in memory until each trace is complete.

Let's go back to the slow checkout problem and see how the three pillars fit together.

Metrics told you something was wrong. Traces told you where. And logs told you why.

That investigation only worked because the three pillars were connected.

The most important link is the trace ID. If every log line includes the trace ID of the request that produced it, you can jump from a trace directly to the relevant logs.

Many systems also attach example trace IDs to metrics. These are called exemplars. So when you see a latency spike on a chart, you can click through to a trace of a request that was part of that spike.

Without these links, you end up with three separate tools and a lot of manual guesswork during an incident, matching timestamps by hand across thousands of concurrent requests.

All this data has to come from somewhere. The process of adding code that produces metrics, logs, and traces is called instrumentation.

Libraries for HTTP servers, database clients, and message queues can create many spans and metrics automatically, while business metrics such as orders placed per minute are added by hand.

For a long time, each monitoring vendor had its own libraries and data formats, which made switching tools painful.

OpenTelemetry is an open-source project that addresses this. It provides a vendor-neutral set of APIs, SDKs, and a data format for generating and exporting metrics, logs, and traces.

Applications send telemetry to an OpenTelemetry Collector, which forwards it to backends such as Prometheus for metrics, Loki or Elasticsearch for logs, and Jaeger for traces.

This way, the instrumentation in your code does not have to change when your backend does.

A good rule of thumb when instrumenting a service: if this breaks during an incident, what will I need to know to fix it? Then make sure the system captures that information by default.

Observability is the ability to understand what is happening inside a system from the data it produces.

Each one has its own costs and limits. Collecting all three does not make a system observable on its own. The value comes from connecting them, through trace IDs in logs and exemplars on metrics, so you can move from an alert to a trace to the exact log line that explains the problem.