When you need to store a terabyte of data, a single hard drive works fine. When you need to store a petabyte, you have a problem. No single machine can hold that much data, and even if it could, you would have a single point of failure that could take down your entire system.

Distributed file systems solve this by spreading data across many machines while presenting a unified view to applications.

When your application reads /data/logs/server.log, the file system figures out which machines hold pieces of that file, fetches them, and assembles them transparently. Your application sees a normal file, but behind the scenes, dozens or hundreds of servers may be involved.

Distributed file systems also provide fault tolerance (your data survives machine failures), high throughput (many machines serving data in parallel), and data locality (processing can happen where data lives). These properties make distributed file systems the backbone of modern data infrastructure, from web search indexes to machine learning training pipelines.

In this chapter, we will explore how distributed file systems work, the architectural patterns they use, and when to choose them for your system design.

What is a Distributed File System?