Log aggregation collects logs from many sources into a central pipeline so engineers can search, correlate, retain, protect, and route them consistently.

In distributed systems, logs come from services, containers, serverless functions, queues, databases, proxies, batch jobs, and third-party integrations. If those logs stay on local disks or isolated tools, evidence disappears or becomes painful to retrieve.

In this chapter, you will learn how log aggregation pipelines collect, process, store, query, protect, and monitor logs across distributed systems.

On a single server, local logs are enough for simple debugging. On a distributed system, local logs become a liability.

A single checkout failure may involve gateway logs, application logs from several services, queue producer and consumer logs, database slow query logs, load balancer logs, model gateway or third-party provider logs, and deployment or configuration change logs.

Without aggregation, the responder must know where each log lives, have access to every host or tool, and manually line up timestamps. That is slow on a good day and unreliable during an incident.

Why Local Logs Fail

Local logs fail because storage is scattered, infrastructure is ephemeral, production access becomes risky, related events are hard to correlate, retention differs by system, redaction and audit policies are uneven, and responders spend time finding logs instead of understanding the incident.

The goal is not to keep every byte forever. The goal is to preserve the right evidence, make it searchable, and apply consistent controls.