Premium Video

This video is available to premium subscribers only

Unlock Full Access

Reading events from a Kafka topic is the easy part of stream processing. The hard part is producing correct results when the work is spread across many machines, machines crash, events arrive late, and traffic spikes faster than the pipeline can keep up. Stream processing engines exist so that each team doesn't have to solve those problems from scratch.

In this chapter, we will look at what a stream processing engine does, how it distributes work across multiple machines, how it manages state and recovers from failures, and how popular engines like Flink, Kafka Streams, and Spark Structured Streaming compare.