Lambda Architecture solves the latency-accuracy trade-off by running batch and stream processing in parallel. It works, but maintaining two separate codebases for the same logic is painful. Every change requires updates to both systems. Testing is complex. Bugs can cause the two implementations to diverge.

In 2014, Jay Kreps (co-creator of Apache Kafka) proposed a simpler alternative: Kappa Architecture. The core idea is radical. Instead of running batch and stream in parallel, use only stream processing. If you need to reprocess historical data, replay the event log through the same streaming system.

Kappa Architecture trades the complexity of dual systems for the challenge of making stream processing robust enough to handle all use cases. With modern streaming engines and event logs, this is increasingly practical.

In this chapter, you will learn:

The Core Idea

Kappa Architecture takes the messy “two pipelines” problem and replaces it with a single, consistent path:

The Key Insight

Everything is a stream.

Batch processing is just a special case. It is a stream with an end. If your stream processor can handle a continuous stream, it can also handle a bounded stream.

Kappa works when you have three things:

If those conditions hold, you do not need a separate batch layer. Reprocessing is simply replaying history through the same pipeline.

Reprocessing in Kappa

Reprocessing is the moment Kappa becomes real.

When you find a bug, change business rules, or want to add a new derived field, you do not spin up a separate batch job. You do a controlled cutover.

The common pattern

Why this is powerful

The cost is that replay can be expensive if your history is huge and your pipeline is heavy.

Lambda vs Kappa

Architecture Comparison

Detailed Comparison

When Lambda Still Wins

Kappa is not automatically better. There are situations where Lambda is still the more practical choice.

When Kappa Wins

Kappa shines when simplicity and speed of iteration matter, and when your system is already event-driven.

The Event Log

The event log is the foundation of Kappa. It is the system of record.

Your views can be deleted and rebuilt. Your processors can be upgraded. Your source of truth remains the log.

Requirements for the Event Log

For true Kappa, you typically want long retention, often “effectively infinite” through tiered storage.

Apache Kafka as Event Log

Kafka is the most common implementation because it supports the event-log model naturally:

It also supports multiple consumers at once, for example:

Retention Strategies

Retention is a design decision. It shapes what kind of “Kappa” you can realistically support.

For true Kappa Architecture, you need long or infinite retention.

Tiered Storage

Tiered storage makes long retention practical by splitting data across cost tiers:

Modern Kafka supports tiered storage:

You still get the same log abstraction. Consumers can replay old data even if it lives in the cold tier. You get reprocessing capability without paying premium storage costs forever.

Reprocessing in Practice

Kappa sounds simple on paper: replay the log, rebuild the views, move on. In practice, the trick is doing it without breaking production, confusing users, or losing confidence in your numbers.

The good news is that the workflow is well understood and repeatable.

The Reprocessing Workflow

Think of reprocessing as a controlled migration from v1 to v2 of your streaming job and its output views.

Phase 1: Normal processing

Phase 2: Deploy new version

At this moment, you are running two pipelines:

Phase 3: Catch up

Phase 4: Cutover

Phase 5: Decommission

The key principle: never mutate the old view in place. Build the new one side-by-side, then switch.

Challenges with Reprocessing

A practical mindset: during reprocessing, you are intentionally running in a “migration mode.” You plan for temporary cost and temporary complexity, but you make the cutover clean.

Parallel Reprocessing

Reprocessing speed is often limited by how fast you can read and process the log. Partitioning is your friend.

If the event log has 100 partitions, you can assign them across multiple instances of the new job:

All instances write into the same New View, usually through a sink that can handle concurrent writers safely.

Kappa Architecture Implementation

A complete Kappa setup typically has these pieces:

1. Event log

Kafka (or similar) with long retention. This is your system of record.

2. Stream processing engine

Flink, Kafka Streams, or Spark Streaming, often doing stateful processing and windowing.

3. Serving views

Stores optimized for query patterns:

4. Cache layer

Optional but common, especially for dashboards and APIs that need very fast reads.

5. API and dashboards

The application queries the serving views, not the event log.

Technology Stack

Stream Processing Requirements

Kappa moves the burden of correctness to streaming. If your stream processing is flimsy, your entire architecture becomes fragile.

A practical Kappa-capable processor needs:

Handling Edge Cases

Late Arriving Events

Events can arrive out of order. How do you handle an event from 2 hours ago?

Common strategies:

Which one you choose depends on what “correctness” means for your product. Dashboards often allow corrections. Billing systems often require strict cutoffs and explicit adjustments.

Windowed Aggregations

For queries like "events per hour," define windows:

Common window types:

Window choice is not just a technical detail. It directly affects how users interpret the metric.

State Management

Stream processors maintain state for aggregations:

The typical pattern:

This is what makes long-running streaming jobs reliable. Without state plus checkpointing, you either lose accuracy or you are forced to replay huge ranges after every incident.

Kappa vs Lambda: Making the Choice

Choosing between Kappa and Lambda is not about picking the “modern” one. It is about matching the architecture to your constraints.

A good decision comes down to two questions:

If the answer to the second question is uncertain, Lambda often becomes the safer choice. If the answer is clearly yes, Kappa usually wins on simplicity.

Decision Framework

Summary Comparison

Summary

Kappa Architecture simplifies data processing by using only stream processing: