A strong offline score does not guarantee production success. It only says the model performed well under the assumptions of the evaluation setup. When those assumptions are wrong, the score measures nothing you can ship against.

The failures behind this are mundane and common. A preprocessing step is fit on the full dataset before the split. The same user lands in both train and test. A feature uses information that would not exist at prediction time. A test set reflects last quarter's traffic while the model serves today's users.

Evaluation metrics tell you what to measure. This chapter covers how to measure it correctly: splitting data, preserving time, avoiding leakage, and building evaluation sets that resemble the serving environment.