When a monolith misbehaves, you usually have one process, one set of logs, and one stack trace to inspect. That is often enough to find the problem.

In a distributed system, one request may pass through dozens of services before it returns. A health check can say everything is "green" while a real user is still waiting, seeing an error, or getting a different result than everyone else.

Observability closes that gap. It means understanding what is happening inside a running system by looking at the signals it produces: logs, metrics, and traces.

This chapter explains how those three signals work together, especially when basic monitoring tells you something is wrong but does not explain why.

In a monolithic application, debugging is usually straightforward. You have one process, one log file, and one stack trace when something goes wrong. You can often reproduce the issue locally and step through the code.

Distributed systems are different. A single user request might touch many services, each running on different machines, with its own logs and failure modes. The problem might be in one service, the network between services, or the way two healthy services interact.

When latency spikes in this system, where do you look first? The database? The external API? The network between services? A slow cache lookup?

Without observability, finding the answer requires guesswork and luck.

The Debugging Challenge

The usual debugging tricks do not go far enough in distributed systems. A local debugger cannot follow a live request across production services. Print statements become noise under heavy traffic. Local log files stop helping once work is spread across many servers. And local reproduction often misses timing bugs, network issues, and problems that only appear under load.

Observability tools fill this gap. They collect signals from across the system and connect them together, so you can ask useful questions:

Observability rests on three pillars: logs, metrics, and traces. Each one answers a different question. Together, they give you a clear view of what is happening inside your system.

Logs tell you what happened in individual events.

Metrics tell you how often something happened, how long it took, or how much of a resource is being used.

Traces show the path one request took across services and where it spent time.

A simple way to remember them is to think of a doctor's toolkit: logs are clinical notes, metrics are vital signs, and traces are scans that show paths and connections.

Logs are timestamped records of events in your system. Whenever something important happens, your code writes a log entry that describes it.

What Logs Look Like

A few lines from an order flow show the usual shape: a timestamp, a level, the service name, and the event with useful context.

Each entry should tell you when the event happened, where it happened, what happened, and the key details you may need later, such as order_id, user_id, or the error message.

Structured vs Unstructured Logs

Modern systems usually prefer structured logs (often JSON) over plain text. They are easier for log tools to read, search, and filter.

Unstructured (hard to search reliably)

Written as a sentence, the event is easy for a person to read but awkward for software to search. The log system has to guess where each value is.

Structured (machine-readable)

The same event in JSON puts every value under a clear field name. Now the log system can search for order_id, user_id, or total directly.

With structured logs, queries become straightforward: show all ERROR logs from payment-service in the last hour, find failed orders over $1000, or count events by type for each service.

What Logs Are Good For

Logs are especially useful when you need detail. They help you debug errors, keep records of important actions, investigate security events, and understand why the system made a specific decision.

Limitations of Logs

Logs are essential, but they are not enough by themselves. A busy system can produce billions of log lines per day. Storage and search get expensive. Useful entries get buried in noise. Logs also describe individual events, so they are not great for seeing trends. And one service's log rarely explains the whole distributed request.

This is why logs alone are not enough.

Metrics are numbers collected over time. While logs capture individual events, metrics summarize many events into time series, which are values tracked minute by minute or second by second. This makes it easy to spot trends, compare behavior over time, and catch problems early.

What Metrics Look Like

The samples below show three common metric shapes: a counter that only goes up, a gauge that can go up or down, and a histogram that groups values into buckets.

Types of Metrics

Metrics generally fall into a few common types:

Counters

Counters track totals that only go up, such as requests served, errors, or bytes transferred.

Gauges

Gauges track current values that can go up or down, such as active connections, queue depth, or memory usage.

Histograms

Histograms group measurements into buckets, such as request latency, request size, or processing duration.

Here’s a quick summary:

The Four Golden Signals

If you track nothing else, track these four. They cover the most common ways services become slow, overloaded, or unreliable.

Latency shows how long requests take. Traffic shows demand. Errors show how often requests fail. Saturation shows how close a resource is to its limit.

They quickly tell you whether your service is healthy and whether it is trending toward trouble.

What Metrics Are Good For

Metrics are useful for alerts, capacity planning, SLA tracking, spotting unusual behavior, and dashboards because they turn many events into trends you can graph and compare.

Limitations of Metrics

Metrics are efficient, but they have blind spots. Summaries hide details. Too many unique labels, such as user_id or request_id, can overload your metrics backend. Error spikes do not show which users were affected. And you can only ask questions about the metrics you decided to collect ahead of time.

Metrics usually tell you that something is wrong. To find where it went wrong in a distributed system, you need traces.

Traces follow one request as it moves through your distributed system. They show which services were called, in what order, and how long each step took. If metrics tell you something is wrong, traces help you find where it went wrong.

What Traces Look Like

This trace breaks a single 450ms request into the services it touched. Each node shows how much time that step took.

A trace is made up of spans. A span is one piece of work, such as a service call, a database query, or a cache lookup.

What a span typically includes

A span usually includes a trace ID, span ID, parent span ID, operation name, start time, end time, and key-value fields that describe the work.

What Traces Are Good For

Traces are useful for finding bottlenecks, understanding which services depend on each other, debugging specific requests, spotting failures that spread across services, and improving the slowest parts of important flows.

Limitations of Traces

Traces are powerful, but they have tradeoffs. Storing every trace is expensive, so most systems keep only a sample. Every service must pass trace context to the next service. Traces also cost more to store than metrics, and very large traces with hundreds of spans can be hard to read.

The real power of observability comes from combining metrics, traces, and logs. Each pillar answers a different question. Together, they take you from "something is wrong" to the most likely cause.

A practical debugging workflow

Consider a case where users report slow checkout. The value of the three pillars is not just that each one answers a different question. The real value is that you can move from one to the next using shared IDs:

The shared trace_id, and the metric exemplar that points to it, turns three separate data sources into one investigation. Without that link, you end up matching timestamps by hand. With it, you can move from symptom to cause quickly.

These terms are often used as if they mean the same thing, but they solve different problems.

Monitoring is about watching for known problems. You decide ahead of time what to measure, set limits, and alert when something crosses the line. This works well for problems you can predict.

Observability is about investigating unknown problems. You collect enough useful signals, including logs, metrics, traces, and context, so you can ask new questions when something unexpected happens. This matters in distributed systems because real failures rarely follow a neat script.

Quick comparison

The table shows the practical difference:

In practice, you need both. Monitoring catches the issues you can predict. Observability helps you debug the ones you cannot.

Observability does not happen by accident. You have to design for it, just like scalability or reliability.

Instrument Everything

Every service should produce logs, metrics, and traces as a normal part of the codebase.

Use Consistent Standards

Observability breaks down quickly when every team does things differently. Use consistent JSON fields for logs. Give metrics clear names and avoid labels with unlimited unique values. Use stable span fields for traces, and follow standard ways of passing trace IDs between services, such as W3C Trace Context.

Connect the Pillars

The pillars are most useful when you can move between them quickly. Include trace IDs in log entries and attach exemplars, or example trace links, to metric points when your tools support it.

This lets you jump from a metric spike to the traces behind it, then to the logs for the slow or failing span.

Design for Debuggability

A good rule of thumb is simple: If this breaks during an incident, what will I need to know to fix it?

Then make sure the system captures that information by default.

Observability is about understanding what is happening inside a running system. Logs record individual events with useful context, metrics turn behavior into numbers over time, and traces follow requests across services.

Each pillar answers a different question: metrics help you notice that something is wrong, traces help you locate where it went wrong, and logs help explain why it happened.

The pillars work best together. A typical debugging flow starts with metrics showing a change, moves to traces to find the slow or failing part, and ends with logs explaining the cause.