In the previous chapter, we explored batch processing as a paradigm for handling large datasets. But understanding the concept is different from understanding how to actually process terabytes of data efficiently.

How do you write a program that runs on thousands of machines simultaneously? How do you handle machine failures when a job takes hours to complete?

MapReduce answered these questions. Published by Google in 2004, it became the foundational model for distributed batch processing.

The insight was deceptively simple: break complex computations into two phases (Map and Reduce) that can be parallelized across thousands of machines, with the framework handling all the messy distributed systems details.

While newer frameworks like Spark have largely superseded MapReduce for many workloads, understanding MapReduce is essential. Its concepts underpin modern data processing systems, and it remains a common topic in system design interviews.

The Problem MapReduce Solves