Log Aggregation
Practice this topic in a realistic system design interview
Suppose your application runs as 20 services, spread across 200 containers. A user reports that their order failed a few minutes ago.
Where do you look?
The request may have touched five different services. Each of those services has dozens of instances, and each instance writes its own logs. You could connect to each machine and search its log files one by one, but that could take hours.
And some of the containers that handled the request may already be gone. When a container is replaced, the logs stored inside it usually disappear with it.
Even if you found all the right files, you would still need to line them up by time to understand what happened.
This is the problem log aggregation solves.
In this chapter, we will learn what log aggregation is, how a log pipeline collects, processes, and stores logs from many machines, and the trade-offs involved in running one at scale.