Lambda Architecture
Last Updated: January 8, 2026
In the first chapter of this section, we explored batch and stream processing as two distinct paradigms. Batch gives you accurate results on complete data but with high latency. Stream gives you fast results but on potentially incomplete data. What if you need both?
Consider a real-time dashboard showing total revenue. Users want to see current numbers updating live, but they also need accurate end-of-day totals. Stream processing can show approximate real-time updates, but late events and edge cases mean it might miss some transactions. Batch processing captures everything but only updates once a day.
Lambda Architecture addresses this by running both systems in parallel. The batch layer provides accuracy. The speed layer provides low latency. The serving layer merges the results. You get the best of both worlds at the cost of maintaining two separate pipelines.
In this chapter, you will learn:
- The three layers of Lambda Architecture
- How batch and speed layers complement each other
- The serving layer and query merging
- Real-world implementations and trade-offs
- When Lambda Architecture makes sense and when it does not
The Problem Lambda Solves
Many real systems need two things at the same time:
- Real-time updates for user-facing experiences
- Accurate historical results for reporting, billing, and audits
That combination is common in revenue, fraud, recommendations, IoT monitoring, and social media metrics.
The Latency-Accuracy Trade-off
If you choose only one paradigm, you usually sacrifice the other.
Stream-only
- Fast results
- Can be approximate
- Late events and edge cases can cause misses or corrections later
Batch-only
- Accurate results
- High latency
- Updates arrive after the batch window completes
| Requirement | Batch | Stream |
|---|---|---|
| Complete data | Yes | Maybe (late events) |
| Exact aggregates | Yes | Approximate |
| Real-time updates | No | Yes |
| Handle reprocessing | Easy | Complex |
Use Cases Requiring Both
| Use Case | Why Both? |
|---|---|
| Revenue dashboards | Live updates + accurate daily totals |
| Fraud detection | Immediate alerts + refined analysis |
| Recommendation engines | Real-time personalization + accurate models |
| IoT analytics | Live monitoring + historical trends |
| Social media metrics | Live counts + accurate engagement stats |
Lambda Architecture Overview
Lambda Architecture was introduced by Nathan Marz in his book "Big Data." It proposes three layers:
- Batch layer computes accurate views from all historical data
- Speed layer computes low-latency views from recent data
- Serving layer answers queries by merging both views
The Three Layers
| Layer | Purpose | Processing | Latency |
|---|---|---|---|
| Batch Layer | Complete, accurate results | Processes all historical data | Hours |
| Speed Layer | Real-time updates | Processes recent events | Seconds |
| Serving Layer | Merge and serve queries | Combines both views | Milliseconds |
The Batch Layer
The batch layer processes the entire dataset to produce accurate views.
Batch Layer Process
- Ingest: New data appends to master dataset
- Process: Batch job runs on entire dataset
- Output: Replace batch views with new results
- Schedule: Repeat on schedule (hourly, daily)
Key Characteristics
| Characteristic | Description |
|---|---|
| Immutable master data | All raw data stored permanently |
| Recompute from scratch | Each run processes entire history |
| Eventually consistent | Results available after job completes |
| Fault tolerant | Can always recompute from raw data |
Example: Daily Revenue Computation
Why Immutable Master Data?
Storing raw, immutable data enables:
- Reprocessing if bugs are found
- New analyses without re-collecting data
- Complete audit trail
- Schema evolution (reprocess with new schema)
The Speed Layer
The speed layer exists for one reason: users do not want to wait for batch.
It processes only the newest data and produces a real-time view that fills the gap until batch catches up.
Speed Layer Process
- Consume: Read events from stream (Kafka)
- Process: Update incremental aggregates
- Store: Write to real-time views
- Expire: Discard after batch catches up
Key Characteristics
| Characteristic | Description |
|---|---|
| Incremental updates | Only processes new events |
| Eventual replacement | Batch layer eventually overwrites |
| Approximate results | May miss late events |
| Low latency | Updates in seconds |
Example: Real-time Revenue Updates
The Speed Layer's Limited Scope
The speed layer only covers recent data:
The speed layer only needs to cover data since the last batch run.
Once the batch layer recomputes and publishes new batch views, the speed layer’s older data becomes redundant and can be cleared.
This keeps the speed layer lightweight.
The Serving Layer
The serving layer makes Lambda usable. Without it, you would have two separate systems and no clean way to query them.
What it does
- Reads from the batch view (last complete batch output)
- Reads from the speed view (changes since that batch)
- Merges the two into a single response
Merge Strategy
For aggregation queries, merging is straightforward:
Serving Layer Data Stores
Serving needs stores optimized for reads. Common choices include:
| Store Type | Use Case | Examples |
|---|---|---|
| Key-value | Simple lookups | Redis, Cassandra |
| Document | Flexible schemas | MongoDB, Elasticsearch |
| Time-series | Metric queries | InfluxDB, TimescaleDB |
| OLAP | Complex aggregations | Druid, Pinot, ClickHouse |
What happens when batch updates
When batch completes, it overwrites the batch view:
Lambda Architecture in Practice
Full Architecture Diagram
A typical deployment looks like this:
Technology Stack Examples
| Component | Option 1 | Option 2 | Option 3 |
|---|---|---|---|
| Ingestion | Kafka | Kinesis | Pulsar |
| Batch storage | HDFS | S3 | ADLS |
| Batch compute | Spark | MapReduce | Flink Batch |
| Stream compute | Flink | Spark Streaming | Kafka Streams |
| Batch serving | Druid | ClickHouse | Cassandra |
| Speed serving | Redis | Memcached | Cassandra |
Challenges and Trade-offs
The Dual Codebase Problem
The biggest criticism of Lambda Architecture: You end up implementing the same business logic twice.
- One implementation for batch
- One for stream
| Challenge | Description |
|---|---|
| Duplicate logic | Same transformation in two codebases |
| Different languages | Batch in Scala, stream in Java |
| Testing complexity | Must verify both produce same results |
| Maintenance burden | Every change requires two updates |
| Skill requirements | Team needs both batch and stream expertise |
Mitigating the Dual Codebase
Some approaches to reduce duplication:
| Approach | How It Helps |
|---|---|
| Unified frameworks | Spark, Flink support both batch and stream |
| Shared libraries | Core logic in reusable modules |
| SQL-based processing | Same SQL for batch and stream |
| Code generation | Generate both from common definition |
Operational Complexity
Running Lambda Architecture requires:
- Maintaining two processing pipelines
- Monitoring both batch and stream jobs
- Managing storage for master data, batch views, and speed views
- Handling failures in each layer independently
- Coordinating batch completion with speed layer reset
When to Use Lambda Architecture
Good Fit
- Mixed latency requirements: Some users need real-time, others need accurate
- Historical reprocessing needed: Bugs require recalculating from scratch
- Complex aggregations: Batch can compute what stream cannot
- High data volumes: Batch efficiently processes massive datasets
- Regulatory compliance: Immutable master data for audit
Poor Fit
- Pure real-time: Speed layer alone might suffice
- Simple aggregations: Stream processing handles it
- Small team: Maintaining two systems is hard
- Low data volume: Overhead not justified
- Tight latency SLAs: Batch layer too slow
Decision Framework
Summary
Lambda Architecture provides both accuracy and low latency by running parallel pipelines:
- Batch layer processes all historical data for accurate, complete results but with high latency.
- Speed layer processes recent events for real-time updates, compensating for batch delay.
- Serving layer merges both views to answer queries with low latency and reasonable accuracy.
- Immutable master data enables reprocessing when bugs are found or schemas evolve.
- Trade-off: Dual codebase maintenance is the main challenge.
- Good fit: Systems needing both real-time updates and accurate historical analysis.
- Evolution: Led to Kappa Architecture, which attempts to simplify by using only stream processing.