Logs tell you what happened to specific requests. But what if you need to know how the system is performing overall? Are response times getting slower? Is the error rate increasing? How many requests are we handling per second?

These questions require metrics: numerical measurements collected over time.

Metrics answer questions like "how many?" and "how fast?" across thousands or millions of requests. They power dashboards, drive alerts, and enable capacity planning. Good instrumentation gives you early warning when things degrade, often before users notice.

In this chapter, you will learn:

This builds on the observability foundation we established earlier. Metrics complement logs by providing the aggregate view that logs cannot offer.

Why Metrics Matter

To understand why metrics are indispensable, consider an e-commerce platform during a flash sale.

What Logs Show:

What Metrics Show

Logs show individual events. To understand the flash sale's impact, you would need to aggregate 50,000 log entries. Metrics give you instant visibility: request rate tripled, latency increased by 40%, error rate is still acceptable, CPU is climbing.

Metrics vs Logs Comparison

Both are essential. Metrics alert you that something is wrong. Logs and traces help you understand why. Think of metrics as the vital signs monitor in a hospital: it tells doctors instantly when something needs attention, but they still need tests and exams to diagnose the cause.

Types of Metrics

There are four fundamental metric types, each designed for different kinds of measurements. Understanding when to use each one is essential for effective instrumentation.

Counters

Counters are monotonically increasing values. They only go up (or reset to zero on restart).

Use for:

Common queries

Examples:

Gauges

Gauges are values that can go up or down. They represent current state.

Use for:

Queries:

Examples:

Histograms

Histograms measure the distribution of values by counting observations in buckets.

Use for:

Queries:

Examples:

Summaries

Summaries are similar to histograms but calculate percentiles on the client side.

The Four Golden Signals

With so many possible metrics to collect, how do you decide what to monitor?

Google's SRE team distilled years of experience into four fundamental metrics that every service should track. If you monitor nothing else, monitor these:

Latency

How long requests take to process.

What to measure:

Why percentiles matter:

Metrics:

Traffic

How much demand is being placed on the system.

What to measure:

Metrics:

Errors

The rate of requests that fail.

What to measure:

Important: Distinguish between client errors (4xx) and server errors (5xx). A spike in 4xx might be a client bug or attack, not a service problem.

Metrics:

Saturation

How "full" the service is and how close to capacity.

What to measure:

Metrics:

Metric Naming Conventions

Good metric names are critical for usability. When you have thousands of metrics across dozens of services, consistent naming is the difference between quickly finding what you need and frustrating searches through documentation.

Prometheus conventions are widely adopted and worth following.

Naming Rules

<namespace>_<name>_<unit>

Examples:

Unit Conventions

Use base units without prefixes:

Labels (Dimensions)

Labels add dimensions to metrics:

Label best practices:

Cardinality: The Hidden Killer

Cardinality is the number of unique time series a metric produces. It is easy to create thousands of series by accident, and once you hit millions, your metrics system slows down, gets expensive, or falls over.

How Cardinality Grows

That is the kind of number that can crash a metrics backend.

Cardinality Guidelines

Dangerous labels (avoid)

Safe labels (usually fine)

Managing High Cardinality

When you truly need per-user or per-request visibility, metrics are usually the wrong tool.

Use logs, not metrics:

Aggregate to lower cardinality:

Instrumenting Your Code

Instrumentation is the practice of adding metrics collection to your code so you can measure how the system behaves in production.

The goal is not to measure everything. It is to measure the parts that matter when you are debugging incidents, tuning performance, or validating business outcomes.

What to Instrument

Instrument these areas first:

Entry Points

Every API endpoint should expose the golden signals: latency, traffic, errors.

Pseudocode:

Dependencies

Every call you do not control should be instrumented for latency and errors.

Database

Cache

External APIs

Resource Pools

Resource saturation is a top cause of cascading failures. Track utilization of anything that can get exhausted.

Connection pool

Thread pool

Business Metrics

Technical metrics tell you whether the system is running. Business metrics tell you whether the product is working.

E-commerce

Payments

These are the metrics that answer: “Are we making money?” and “Is the user journey healthy?” not just “Are servers alive?”

Prometheus Architecture

Prometheus is one of the most widely used open-source metrics systems. If you understand how it works, instrumentation decisions become much easier.

The big picture

A typical Prometheus setup looks like this:

Pull-Based Model

Prometheus uses a pull model. It pulls metrics from services rather than services pushing metrics to a collector.

This changes how you think about discovery, failure detection, and load control.

Why pull is nice in practice

When push can still make sense

Prometheus handles this case with a separate component called a “push gateway,” but the default mental model is still pull.

Metrics Endpoint

Services expose a /metrics endpoint in Prometheus’ text format. Prometheus scrapes it on a schedule and ingests whatever it sees.

A typical endpoint includes HELP and TYPE metadata plus the actual samples:

PromQL Basics

Prometheus Query Language (PromQL) is how you slice and aggregate time series.

Basic queries:

Rate and aggregation:

Counters are totals, so for “per second” you usually want a rate:

Percentiles:

Histograms store bucket counts. PromQL computes percentiles at query time:

This is why histograms are so useful: you can compute p95, p99, and other percentiles across all instances and services without changing your instrumentation.

Metrics Best Practices

Good metrics feel boring in the best way. They are consistent, predictable, and easy to use across services. Most problems with metrics do not come from missing data. They come from messy naming, inconsistent labels, and high cardinality that makes the system expensive and hard to query.

1. Use Consistent Labels Across Services

All services should use the same label names:

2. Separate Technical and Business Metrics

Technical metrics answer: “Is the system healthy?”

Business metrics answer: “Is the product working?”

Keep them separate so each audience gets what they need without mixing concerns.

3. Use Histograms for Latency

Latency is not a single number. The average can look fine while a small percentage of users have a terrible experience. Histograms let you compute percentiles at query time, which is exactly what you want for p95 and p99.

4. Include Status Labels on Duration Metrics

Errors often have very different latency profiles than successful requests.

If you mix them together, your latency charts become misleading.

This makes it possible to answer questions like:

Summary

Metrics give you the quantitative view of system behavior:

The four golden signals are the minimum set every service should have:

Key practices to remember: