Imagine you have implemented structured logging across your services. Each service writes beautiful JSON logs with consistent fields and appropriate context.

There is just one problem: your logs are scattered across 50 servers, each with its own log files. When something goes wrong, you need to SSH into each machine, grep through files, and somehow piece together what happened.

This does not scale. A single request might touch 10 services running on 30 instances. Finding all logs related to that request means searching through 30 different places. During an incident, you do not have time for this.

Log aggregation solves this by collecting logs from everywhere and storing them in a central location. Instead of searching 30 servers, you search one system. Instead of correlating timestamps manually, you filter by request ID and see everything.

In this chapter, you will learn:

This chapter builds on the logging best practices we covered previously. The structured logs you write are only useful if you can search and analyze them at scale.

The Problem with Distributed Logs

In a single-server app, logs live in one place, so debugging is straightforward. In a distributed system, every service writes its own logs on its own machines. Without aggregation, even simple incidents turn into a scavenger hunt.

A single user request might touch:

Now the engineer has to SSH into multiple machines, grep through files, and manually stitch the story together. That does not scale.

The Challenges

Modern infrastructure makes this worse. With containers, auto-scaling, and frequent redeploys, instances are short-lived. When a container crashes, its local logs may vanish. When a node scales down, its files go with it. If you are not centralizing logs, you are losing the evidence you need most during failures.

Log Aggregation Architecture

A log aggregation system centralizes logs so you can search, correlate, and retain them reliably. Most designs break into four layers: collection, processing, storage, and query.

Collection Layer

The collection layer gets logs from sources to the central system:

1) Agents (most common)

Lightweight daemons run on each node, read logs, and forward them.

2) Direct shipping

Applications send logs directly over the network.

Processing Layer

This layer transforms and enriches logs before storage:

Common tools: Logstash, Fluentd pipelines, Vector transforms, custom stream processors.

Storage Layer

This is where logs are indexed for search and kept for retention.

Search-oriented stores

Time-series / columnar approaches

Query Layer

This is how people and systems interact with logs:

The ELK Stack

The ELK stack is a popular setup for centralized logging. It gives you an end-to-end pipeline: collect logs, process them, store them in a searchable index, and explore them through dashboards.

Elasticsearch

Elasticsearch is a distributed search engine built on Apache Lucene. For logs, its strength is fast search over huge volumes of JSON documents.

How it stores logs:

Key concepts:

Example query (errors in payment-service in the last hour)

Tip: use keyword fields (like service.keyword) for exact matches and text fields for full-text search, depending on your mapping.

Logstash

Logstash processes logs through a pipeline:

Input → Filter → Output

Example pipeline configuration

Kibana

Kibana is the UI layer for exploration and visualization:

Alternative Architectures

ELK is a common choice, but it is not the only one. Different stacks make different trade-offs around cost, query flexibility, and operational complexity..

Loki (Grafana)

Loki takes a different approach from Elasticsearch. It indexes only metadata (labels), not the full log content.

The raw logs are stored cheaply (often in object storage), and Loki keeps a lightweight label index so you can quickly narrow down what you want to read.

Elasticsearch vs Loki

Loki is a good choice when:

Loki works best when you keep labels low-cardinality. Put high-cardinality values (request_id, user_id) in the log body, not as labels.

ClickHouse

ClickHouse is a column-oriented database that is excellent for log analytics:

ClickHouse is a strong option when your logging use case is less “grep everything” and more “run analytics at scale” (top errors, percentiles, breakdowns by service, trends over time).

Managed Services

Cloud providers and observability vendors offer managed logging. These reduce the operational burden, but costs can rise quickly at scale.

Managed services are great when you want to move fast, avoid running clusters, or need enterprise features out of the box. The trade-off is usually cost and vendor lock-in.

Scaling Log Aggregation

As log volume grows, your logging pipeline has to scale just like any other production system. The bottlenecks usually show up in three places: ingestion, indexing/storage, and query load.

Volume Challenges

A rough progression looks like this:

The exact breakpoints vary, but the trend is consistent: once you start indexing everything, costs and operational complexity rise quickly.

Scaling Strategies

1. Add a Buffer (Kafka)

Put a durable message queue between collection and processing:

Benefits:

2. Scale Elasticsearch Horizontally

Elasticsearch is designed to scale by adding nodes, but roles matter.

Common node roles

Scaling is not just “add nodes.” You also need to tune shard counts, mappings, refresh intervals, and ingestion rates so the cluster stays stable.

3. Use Hot-Warm-Cold Architecture

Not all logs are equally valuable. Recent logs need fast search. Older logs are rarely queried and can live on cheaper storage.

A common tiering model:

This keeps your “hot” cluster small and fast while still meeting retention and compliance needs.

4. Sampling and Filtering

The cheapest log is the one you never store. Reduce volume before it hits your index.

High-value logs (always keep)

Low-value logs (sample or drop)

A good rule: if you need counts and rates, use metrics. Use logs for context and investigation.

Index Management

In Elasticsearch, index management has a direct impact on cost, stability, and query speed. The core idea is to keep indices easy to delete, easy to query, and sized so shards stay healthy.

Time-Based Indices

A common pattern is to create a new index per day:

Benefits

Index Lifecycle Management (ILM)

ILM automates what you would otherwise do manually: keep recent data fast, move older data to cheaper tiers, and delete it when it expires.

Hot → Warm → Cold → Delete

Typical transitions:

Common ILM actions

Practical tip: even if you do daily indices, rollover-by-size is useful when volume is uneven (for example during incidents or traffic spikes).

Shard Sizing

Shard size matters because shards are the units Elasticsearch moves, recovers, and searches. A useful target for log workloads is typically:

10–50 GB per shard (many teams aim around 20–40 GB)

Rule of thumb: keep shard count under control. A commonly cited guideline is around 20–25 shards per GB of heap per data node, but treat it as a starting point, not a guarantee. The real goal is to avoid a shard explosion that eats memory and slows the cluster down.

Cost Optimization

Log aggregation costs can grow faster than you expect because you pay for volume at every stage: ingestion, processing, indexing, storage, and queries.

The simplest way to control cost is to treat logs as a product with a budget, not an unlimited dump.

Reduce Log Volume

Reducing volume is the highest leverage move because it lowers costs everywhere downstream.

Techniques that work well:

Optimize Storage

Right-Size Infrastructure

Rough cost comparison (1 TB/day)

These are ballpark ranges and can swing widely based on retention, query patterns, and indexing choices.

Costs vary significantly based on retention, query patterns, and feature requirements.

Observability for your Logging System

Your log pipeline is production infrastructure. If it breaks, you lose the evidence you need during incidents, so it must be monitored like any other critical system.

Key Metrics to Track

Collection

Processing

Storage and query

Example alerts

Common Failure Modes

Summary

Log aggregation turns scattered logs into a centralized, queryable system:

Key decisions that drive cost and scale: