Training on a single GPU works up to a point. Then the dataset gets too large, training takes too long, or the model no longer fits in memory.

Distributed training addresses this by spreading computation across multiple GPUs or machines. But it introduces new challenges. You need to handle how gradients are synchronized, how batch sizes impact convergence, and where new bottlenecks appear.

This chapter focuses on how to scale training without breaking performance or efficiency.