When a request fails in production, logs are often the only record of what the system actually did.

But logs only help if people can understand them quickly. If they are inconsistent, missing request details, or full of sensitive data, debugging turns into guesswork.

Good logging records meaningful events from a running system so engineers can piece together what happened. Useful logs have clear context, consistent fields, and just enough detail.

This chapter covers how to choose log levels, add useful context, write structured logs, avoid common mistakes, and keep logging efficient at scale.

Log levels tell readers how important a message is. They help decide what gets stored, what triggers alerts, and what can be ignored during normal operation.

The Standard Levels

Most logging frameworks use the same six levels, ordered from "the system is down" at the top to very detailed debugging at the bottom.

Choosing the Right Level

The most common mistake is overusing ERROR and underusing WARN.

Bad Example: Logging expected conditions as ERROR

Ask yourself: “Should this wake someone up at 2 AM?” If yes, it is usually ERROR or FATAL. If no, it is probably WARN or lower.

Level Selection Guide

A few yes-or-no questions can usually guide you to the right level.

Run new log statements through this path during code review. It keeps levels consistent across the team instead of depending on each author's personal style.

A log message is only useful if it explains what happened. The goal is simple: someone should be able to read one log line and know what it means, what it affected, and what to check next.

The Essential Elements

Every log entry should answer these questions:

Good vs Bad Log Messages

The difference between a helpful log line and a useless one usually comes down to specifics. Good logs name the thing affected, the important values, and the reason.

Bad: vague and unhelpful

These messages state that something went wrong without naming what, where, or why.

Good: specific and useful

The same failures, now with the details an engineer needs during an incident.

The Context Checklist

Before writing a log statement, ask: “If I saw only this log line, would I understand what happened?”

Always include IDs such as user_id, order_id, correlation_id, or request_id when they apply. Add amounts or sizes when they matter. Include status, retry count, error details, timing, and external service names when those details help explain the event.

Including the Right Amount of Context

Too little context makes logs useless:

Too much context creates noise:

Just right:

Include what you need to debug, nothing more.

Structured logging means writing logs as data, usually JSON, instead of plain text sentences.

What Structured Logging Adds

The diagram shows the difference: plain text has to be interpreted, while JSON already has named fields.

With unstructured logs, you treat logs like text files. You write regex rules, maintain a rule for each format, and risk breaking dashboards or alerts after a small wording change.

With structured logs, logs behave like data. Fields are consistent across services, and filtering is much more reliable.

Unstructured log

Written as a plain line, the same order event packs its fields into prose.

Reading this reliably requires regex. Different formats need different regex patterns. Small wording changes can break your log searches.

Structured log

The same event as JSON breaks every value into its own named field.

Now you can search for large orders with event.name=order_placed AND amount>100, all activity for a user with user_id=789, or order service errors with service.name=order-service AND level=ERROR.

Structured Logging Format

Use the same field names across all services:

Implementation Example

Most languages have good structured logging libraries. The idea is the same everywhere: log an event name and attach key-value details.

Once your logs are structured, you can filter, group, and connect events across services without fighting text formats.

One of the biggest logging risks is accidentally exposing sensitive information. Logs get shipped to central systems, copied into tickets, and shared across teams. If a secret lands in logs, assume more people will see it than you intended.

What Not to Log

Never log passwords, API keys, secrets, credit card numbers, government IDs, full session tokens, refresh tokens, JWTs, or personal health information. Treat email addresses, phone numbers, and IP addresses carefully. Depending on your policy, they may need to be masked or left out entirely.

Masking Techniques

When you need to refer to sensitive data for debugging, log the smallest safe piece that still helps you connect related events.

Bad (exposing full values)

Logging the raw card number, email, and key puts secrets into systems that many people can read.

Good (masked for safety)

Masking keeps just enough of each value to recognize it later while hiding the sensitive part.

Automatic Cleanup

Do not rely on every developer remembering to hide sensitive data every time. Add automatic cleanup to the logging path so it happens by default.

Typical controls include blocked fields such as password, token, authorization, and api_key; masks that reveal only safe parts of a value; and pattern checks for secrets hidden inside free-text logs.

Useful patterns include credit card numbers like \b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b, API keys like (api[_-]?key|secret)[=:]\s*\w+, passwords in URLs like password=\w+, and bearer tokens like Bearer\s+\w+.

Logging has a cost. In busy systems, even a small cost per log line can become a real performance and money problem.

The Cost of Logging

Writing a single log line goes through several steps, from formatting the message to writing it somewhere.

A typical log line can trigger string formatting, JSON conversion, memory allocation, and a write to disk or the network. The expensive part is usually I/O, especially when the user request has to wait for it.

Performance Best Practices

Logging adds cost to every request, so a few habits keep it from slowing down important code paths.

Avoid unnecessary work

Do not compute expensive values if the log will not be written.

Bad (always computes expensive data):

Good (only computes if debug is enabled):

Better (let the logging library delay the work when supported):

Use background logging

Direct logging can block request processing while waiting for disk or network.

With direct logging, the request waits while the log is written. With background logging, the request puts the log in a queue and returns while another thread writes it to disk or sends it over the network.

Background logging prevents slow I/O from blocking request processing.

Pick the right production log level

Use DEBUG or TRACE in development, DEBUG in staging, INFO as the normal production default, and temporary scoped DEBUG during incidents.

A common practice is “INFO by default, DEBUG on demand,” with a time limit and scope (specific service, endpoint, or user) so you do not drown in noise.

Sample very frequent events

For events that happen constantly (cache hits, heartbeats), log only a sample.

Random sampling (1%):

Every 1000th event:

If you want accurate counts, do not rely on logs for that. Use metrics (counters, histograms) and keep logs for context.

Rough performance cost

These numbers vary by language, hardware, and logging setup, but they help build intuition. Direct disk logging can add 1-10ms per log because it blocks the request. Background disk logging is often below 0.1ms per log. Direct network logging can add 5-50ms. JSON conversion often costs 0.01-0.1ms. Simple string formatting is usually much cheaper.

At 10,000 requests/sec, even 0.1ms of extra work per request adds up fast. That is roughly 1 second of CPU time per second, just for logging.

The goal is to log smarter, not simply less: correct levels, structured context, background I/O, and sampling where needed.

Avoid these patterns. They are small mistakes in code, but they cost real time during an incident.

Logging Without Context

A message without IDs leaves you guessing. The same message with the affected order, user, host, or field points straight at the problem.

Inconsistent Formats

When every service uses a different timestamp and field layout, searches become painful. Compare the mixed output below with one shared structured format.

Too Much Logging

Logging inside a busy loop can flood the system. The better version logs once before the batch and once after it finishes.

Swallowing Exceptions

Catching an exception and logging only a short message throws away the stack trace. That makes the next engineer guess where the error came from.

Message in One Place, Context Somewhere Else

Putting the only useful description inside an exception message is risky. A caller might catch the exception and log it differently. Log the context where the failure happens, and keep the exception type meaningful.

Avoiding these mistakes costs almost nothing while writing the code. During an incident, the missing context can be the difference between a quick fix and a blind search.

Logs consume disk space. If you do not manage them, they can fill the disk and take your service down.

Rotation Strategies

Rotation means closing the current log file and starting a new one, either on a schedule or when the file gets too large.

The most common strategies are size-based rotation, time-based rotation, or both. For example, you might rotate when a file reaches 100MB, at midnight every day, or whichever happens first.

After rotation, teams often compress old files with gzip and delete them after a set number of days.

Retention Guidelines

Retention means how long you keep logs. It depends on what the logs are used for and whether legal or compliance rules apply.

Tip: keep older logs in cheaper storage, such as object storage, and keep recent logs in the logging system for fast search.

Good logging is designed on purpose. Log levels show importance, context makes entries useful, structured JSON makes logs searchable, automatic cleanup protects sensitive data, background I/O and delayed work keep performance under control, and rotation plus retention stop logs from becoming a reliability problem.

Before adding a log, ask whether it would help during an incident. Prefer structured fields, include request or correlation IDs when useful, avoid sensitive data, and sample very frequent events instead of logging everything.