Premium Video

This video is available to premium subscribers only

Unlock Full Access

Suppose you get paged at 2 a.m. because payments are failing. You open the logs for the payment service and search for errors.

You find thousands of lines that say Something went wrong. No order ID. No user ID. No indication of which downstream call failed.

Now imagine the opposite problem. The service logs every function call, every cache lookup, and every health check. The one error you need is buried under millions of lines of noise.

Both situations leave you with logs that are hard to use when you need them.

In this chapter, we will look at how to write logs that are useful in production, covering structure, log levels, context, what to log and what to leave out, and how to keep logging costs under control.

A useful log entry should answer a few basic questions:

Take a log line that says:

It tells you what happened, but nothing else. Now compare it to this:

The second log gives you enough to start investigating without opening the code. The rest of this chapter is about how to produce logs like the second one, consistently, across every service.

A good test before writing any log statement: if I saw only this line during an incident, would I understand what happened?

The first practice is to use structured logs.

Many applications write logs as plain sentences, with values mixed into the text:

But once you have millions of log lines from dozens of services, you need to search and filter them with a machine. Pulling values out of sentences means writing regex rules, and a small wording change can break every search and alert built on them.

Structured logging writes each entry as a set of named fields, most commonly in JSON. A structured log for the failed payment might look like this:

Now your log system can index those fields. You can ask for every payment timeout in the last hour (service=payment-service AND error_type=timeout). You can group errors by type. You can find every log entry for one specific order (order_id=842).

Consistent Field Names

Structured logs only help if the fields are consistent.

Suppose one service logs the user as user_id. Another logs it as userId. A third uses uid. Now a single search for one user's activity has to know every variation.

So teams usually agree on a shared logging schema. It defines standard names for common fields:

Many teams enforce this through a shared logging library, so every service produces the same field names without each developer having to remember them.

The next practice is using log levels correctly. Most logging libraries support a similar set of levels.

Some libraries also have FATAL above ERROR and TRACE below DEBUG, but the four in the middle cover almost every case.

Levels let you control how much detail you keep. In production, many teams log at INFO and above, and only turn on DEBUG for a specific service when they are investigating something.

Choosing the Right Level

Picking the wrong level causes real problems.

Suppose a user types the wrong password. That is not an error in your system. It is an expected part of normal operation.

If you log it at ERROR level, your error count grows every time someone mistypes a password. Alerts built on error logs start firing for things that need no action. Over time, the team learns to ignore those alerts, and real errors get missed.

A useful rule is to ask whether someone needs to act.

If yes, it is probably an error. If it is unusual but handled, it is probably a warning. If it is part of normal flow, it is info or debug.

A log message on its own is rarely enough. It needs context.

Some context is the same for every log a service writes. This includes the service name, the version being deployed, the environment, and the host or container it runs on.

The version is especially useful. If errors start right after a deployment, the version field lets you confirm that only the new version is affected.

Other context depends on the request. This includes the request ID, the user ID, and the main entity being processed, such as an order ID.

The first four fields are added once, when the logger is created. The next three are added per request. The developer only writes the last two.

When you log an exception, include the error type, the message, and the stack trace. A log that says database error is far less useful than one that shows the exception, the query that failed, and the line of code where it happened.

In a distributed system, one request can create log entries in many services. To connect them, every log entry for that request needs to share an identifier. This is called a correlation ID or request ID.

Now you can search for one ID and see the full story of a request across all services, in order.

If you use distributed tracing, the trace ID can serve as this identifier, which also links your logs to your traces.

Most logging libraries can store this ID in a request-scoped context and add it to every entry automatically, so developers do not have to pass it through every function call:

What should you actually log? A good starting point is the boundaries of your system.

Just as important is knowing what to keep out of logs.

Logs are often less protected than your database. They get copied into log aggregation systems, backups, and third-party tools, and many engineers may have access to them. A secret written to a log can end up in far more places than the original data.

Regulations such as GDPR and PCI DSS also place rules on how personal and payment data is stored, and logs are covered too.

The safest approach is to redact sensitive fields in the logging library itself. For example, the logger can mask any field named password or token, or show only the last four digits of a card number.

That way, protection does not depend on every developer remembering it every time.

More logs do not always mean better visibility.

Logging Inside Tight Loops

One common source of noise is logging inside tight loops. If a job processes one million records and logs each one at INFO level, you get one million lines that tell you very little.

It is usually better to log a summary, such as how many records were processed, how many failed, and how long it took.

Logging the Same Error at Every Layer

Another source of noise is logging the same error at every layer.

Suppose the database layer catches an exception, logs it, and rethrows it. The service layer catches it, logs it, and rethrows it. Then the controller logs it again. Now one failure appears as three errors, which makes error counts misleading.

A good rule is to log an error once, at the place where it is actually handled.

The message itself also matters. Keep the message short and constant, and put the changing values into fields.

For example, the message might always be Payment charge failed, with the order ID, amount, and error type stored as separate fields.

This makes it easy to count how often that exact event happens, because every occurrence has the same message. If you build the message by joining strings together, every line looks different, and grouping them becomes much harder.

Logging is not free. Every log line uses CPU to format, memory to buffer, and disk or network bandwidth to write. In a high-traffic service, careless logging can noticeably slow down requests.

Asynchronous Logging

One common fix is asynchronous logging. Instead of writing to disk or the network on the request thread, the application puts log entries into an in-memory buffer, and a background thread writes them out.

This keeps slow disks or network hiccups from blocking user requests.

The trade-off is that entries still in the buffer can be lost if the process crashes, and the buffer needs a policy for what happens when it fills up: block the caller, or drop entries (often dropping DEBUG and INFO first).

Do Not Build Messages You Will Not Use

Another fix is to avoid building expensive debug messages when debug logging is turned off. Most logging libraries check the level first, so a disabled debug message costs almost nothing, as long as you do not compute its values in advance.

Bad (always serializes, even when DEBUG is off):

Good (placeholder, no string building if DEBUG is off):

Better (Log4j 2 supplier: the expensive work runs only if DEBUG is on):

To build intuition: at 10,000 requests per second, even 0.1 ms of extra logging work per request adds up to roughly 1 second of CPU time every second.

In modern deployments, logs should not stay on the machine that produced them. Containers are replaced constantly. When a container is removed, any log files stored inside it are gone.

So the common pattern is:

There, logs from every service are stored together, indexed, and searchable in one place.

Writing to standard output also keeps the application simple. It does not need to know where logs end up, and it does not need to manage log files, rotation, or disk space.

As systems grow, log volume can grow into terabytes per day, and so does the bill. There are a few common ways to manage this.

Sampling

Keep every error and warning, but only a fraction of successful request logs, for example 10%.

Changing Log Levels at Runtime

Some systems let you raise a single service to DEBUG level for a short time without redeploying, and then lower it again. This gives you detail exactly when and where you need it, without paying for it all the time.

Retention Tiers

Recent logs are kept in fast, searchable storage for a week or a month. Older logs are moved to cheaper storage such as object storage, or deleted based on your compliance requirements.

Use Metrics for Counting

Finally, do not use logs for things metrics handle better. If you want to know how many requests you serve per second, count them with a metric instead of writing a log line for each request and counting later.