During an incident, the logs you need are rarely in one place. They may be spread across services, containers, servers, queues, and serverless functions. Some may disappear when a container restarts. Searching machine by machine while users are affected is not a real plan.

Log aggregation solves this by collecting logs from every source into one central system. Once logs are in one place, you can search them, connect events across services, store them for the right amount of time, protect sensitive data, and monitor the logging system itself.

This chapter explains how log aggregation works, how logs move through the pipeline, where they are stored, and what design choices keep the system useful, safe, and affordable.

On a single server, local logs can be enough for simple debugging. In a distributed system, local logs quickly become a problem.

A single checkout failure may involve gateway logs, application logs from several services, queue logs, database slow-query logs, load balancer logs, third-party provider logs, and deployment or configuration-change logs.

Without aggregation, the engineer responding to the incident has to know where every log lives, have access to every host or tool, and line up timestamps by hand. That is slow on a good day and painful during an incident.

Why Local Logs Fail

Local logs fail for practical reasons. Storage is scattered. Containers and tasks can disappear. Production access is risky. Related events are hard to connect. Retention differs from system to system. Sensitive-data rules are uneven. Engineers spend time finding logs instead of understanding the incident.

The goal is not to keep every byte forever. The goal is to keep the right evidence, make it searchable, and apply the same safety rules everywhere.