A model can report 99%+ accuracy and still fail at its job. In imbalanced problems like content moderation, a model that predicts the majority class most of the time can look great on paper while missing the cases that actually matter.

This is what happens when you measure the wrong thing.

Evaluation metrics are the lens through which you judge model quality. Pick the wrong one, and a weak model can appear strong.

This chapter covers the three main families of metrics in ML systems: classification, regression, and ranking. More importantly, it focuses on how to choose the right metric and what to do when they point in different directions.

Classification models predict discrete categories. The foundation for evaluating them is the confusion matrix, a 2x2 table (for binary classification) that counts how predictions map to reality.

Every classification metric is derived from these four cells. The choice of metric determines which cells you care about, and that choice should be driven by the cost of each error type in your specific system.

The Accuracy Trap

Accuracy measures overall correctness: (TP + TN) / Total. It works well when classes are roughly balanced. A sentiment classifier on a dataset with 52% positive and 48% negative reviews can meaningfully report accuracy because getting either class wrong is equally likely to affect the score.

But accuracy falls apart on imbalanced data. In fraud detection, where 0.1% of transactions are fraudulent, a model that never flags anything scores 99.9% accuracy. In medical screening, where 2% of patients have a condition, a model that always says "healthy" is 98% accurate. The metric rewards ignoring the minority class, which is usually the class you care about.

Precision and Recall

Precision answers "of everything I flagged, how much was actually positive?" and recall answers "of everything that was actually positive, how much did I catch?" These two metrics capture the two distinct failure modes of a classifier.

Precision = TP / (TP + FP). High precision means few false alarms. When your spam filter flags an email as spam, precision measures how often it's right. Low precision means legitimate emails end up in the spam folder.

Recall = TP / (TP + FN). High recall means you're catching most of the positives. In fraud detection, recall measures what fraction of actual fraud the model identifies. Low recall means fraudulent transactions slip through.

The tension between them is fundamental. Lowering the classification threshold catches more positives (higher recall) but also triggers more false alarms (lower precision). The right operating point depends on the business cost of each error type, not on any mathematical rule.

F1-Score and Its Variants

The F1-score combines precision and recall into a single number: F1 = 2 (Precision Recall) / (Precision + Recall). It uses the harmonic mean rather than the arithmetic mean because the harmonic mean penalizes imbalance between the two values. A model with 95% precision and 10% recall gets an arithmetic mean of 52.5% (sounds okay) but an F1 of 18.2% (correctly reflects that the model barely catches anything).

F1 assumes precision and recall matter equally. In practice, they rarely do. If missing fraud costs $10,000 per incident and a false alarm costs $5 in manual review, you'd gladly sacrifice precision for recall. F1 can't express that preference. Use it as a quick summary, but don't optimize for it blindly.

For multi-class problems, there are three aggregation strategies:

AUC-ROC

A single precision or recall number depends on the classification threshold. Change the threshold, and both metrics shift. AUC-ROC provides a threshold-independent evaluation by measuring performance across all possible thresholds.

The ROC curve plots the True Positive Rate (recall) against the False Positive Rate (FP / (FP + TN)) at every threshold. AUC-ROC is the area under this curve. A perfect model has AUC-ROC = 1.0. A random model has AUC-ROC = 0.5.

The intuitive interpretation: AUC-ROC is the probability that the model ranks a random positive example higher than a random negative example. An AUC of 0.92 means that 92% of the time, a randomly chosen fraudulent transaction gets a higher score than a randomly chosen legitimate one.

AUC-ROC works well when both classes matter roughly equally. But on heavily imbalanced datasets, it can be misleading. A fraud detection model on a dataset with 0.1% fraud might achieve AUC-ROC of 0.98 because correctly ranking the 99.9% negatives is easy and dominates the score. The model might still perform poorly on the tiny positive class, but AUC-ROC won't reveal that.

AUC-PR

The Precision-Recall (PR) curve plots precision against recall at every threshold. AUC-PR is the area under this curve. Unlike ROC, the PR curve focuses entirely on the positive class, making it more informative when the positive class is rare and the one you care about.

Consider a fraud detection model evaluated on 1 million transactions where 1,000 are fraudulent (0.1% fraud rate).

The AUC-ROC of 0.98 says the model ranks most fraud above most legitimate transactions. That sounds great. But the AUC-PR of 0.45 reveals that when you look at the items the model ranks highest, a large fraction are false positives. The model's actual ability to surface fraud at the top of its ranking is mediocre.

For any system where the positive class is rare and important (fraud, abuse, rare disease), AUC-PR gives you a more honest picture than AUC-ROC.

Regression models predict continuous values: delivery time, stock price, temperature, user lifetime value. Evaluating them requires measuring "how far off" the predictions are, but different metrics define "how far off" in different ways.

Consider two ETA prediction models for a ride-hailing app, both predicting trip duration in minutes:

Which is better? That depends entirely on which metric you use and what the business values.

MAE, MSE, and RMSE

MAE (Mean Absolute Error) is the most intuitive regression metric. It averages the absolute difference between predictions and actuals. If MAE is 2.5 minutes, the model is off by 2.5 minutes on average. Every error contributes proportionally: a 10-minute error counts exactly twice as much as a 5-minute error.

MSE (Mean Squared Error) squares each error before averaging. This disproportionately penalizes large errors: a 10-minute error contributes 100 to MSE, while a 5-minute error contributes only 25. MSE treats a single 10-minute error as worse than two 5-minute errors.

The classification chapter covered MSE and MAE as training loss functions. As evaluation metrics, the same formulas serve a different purpose: they define what "good enough" means for your users, not what the optimizer minimizes during training. You might train with MSE (because it's smooth and differentiable) but evaluate with MAE (because it's what users experience).

RMSE (Root Mean Squared Error) is the square root of MSE, which brings the units back to the original scale (minutes, dollars, degrees). RMSE is always greater than or equal to MAE. The gap between RMSE and MAE tells you something useful: a large gap means the error distribution has high variance, indicating some predictions are much worse than others.

Back to the ETA models: Model A has lower MAE (2.5 vs 3.0) but higher RMSE (6.8 vs 4.2). Model A is better on average but has large outlier errors. Model B is slightly worse on average but more consistent. For a ride-hailing app, users might tolerate being 3 minutes off consistently but get frustrated when the app says "5 minutes" and the ride takes 20. RMSE captures that frustration better than MAE.

R-squared

R-squared (R2) measures how much variance in the target the model explains. R2 = 1 means the model captures all variance perfectly. R2 = 0 means the model is no better than predicting the mean for every input. R2 can go negative, which means the model is actively worse than just guessing the average.

R-squared is useful for comparing models on the same dataset ("Model A explains 85% of variance, Model B explains 72%"). It's less useful for communicating to stakeholders because it doesn't tell you the actual error magnitude. An R2 of 0.90 on house prices could mean errors of $10,000 or $100,000 depending on the variance of the dataset.

MAPE

MAPE (Mean Absolute Percentage Error) expresses error as a percentage of the actual value. If MAPE is 12%, the model is off by 12% on average. This makes it easy to communicate to business stakeholders: "Our demand forecast is accurate to within 12%" is more intuitive than "MAE is 340 units."

MAPE has two gotchas. First, it's undefined when the actual value is zero (division by zero). Demand forecasting for a new product with zero historical sales breaks MAPE. Second, it's asymmetric: overestimating by 50% (predict 150 when actual is 100) gives a MAPE of 50%, but underestimating by 50% (predict 50 when actual is 100) also gives 50%. However, predicting 200 when actual is 100 gives 100%, while predicting 0 when actual is 100 gives 100%. The scale of penalties isn't symmetric in practice, especially near zero. Symmetric MAPE (SMAPE) and weighted MAPE variants exist to address this, though they introduce their own quirks.

Recommendation systems and search engines don't output a single prediction. They produce an ordered list. A search engine returns 10 blue links. A recommendation feed shows 20 videos. The quality of these systems depends not just on whether the right items appear, but on where they appear in the list.

A model with high AUC-ROC can still produce bad rankings. AUC measures whether the model scores relevant items higher than irrelevant ones overall, but it doesn't care about position. A user who scrolls past 8 irrelevant results before finding something useful has a bad experience, even if the model's aggregate classification accuracy is high. The first result in Google Search gets roughly 30% of all clicks. The tenth result gets about 2%. Position matters enormously.

Precision@k and Recall@k

Precision@k measures the fraction of the top k results that are relevant. If you show 10 recommendations and 7 are relevant, precision@10 = 0.7. It answers a simple question: "When we show the user a list, how much of it is useful?"

Recall@k measures the fraction of all relevant items that appear in the top k. If there are 50 relevant videos in the catalog and 7 appear in your top-10 list, recall@10 = 0.14. It answers: "How much of the relevant content are we surfacing?"

Increasing k naturally improves recall@k (more chances to include relevant items) but often decreases precision@k (more irrelevant items sneak in). The right k depends on the product: a search results page might show k=10, a mobile recommendation feed might show k=5 before the user scrolls.

MRR (Mean Reciprocal Rank)

MRR focuses on a single question: where does the first relevant result appear? For each query, compute 1/rank_of_first_relevant_result, then average across all queries. If the first relevant result is at position 1, the reciprocal rank is 1.0. At position 3, it's 0.33. At position 10, it's 0.1.

MRR works well for navigational queries where there's essentially one right answer: "What's the capital of France?" or "Nike Air Max official website." The user wants the answer at position 1, and every position below that is progressively worse.

MRR is a poor fit for exploratory queries where multiple results are valuable: "best restaurants in Tokyo" or "machine learning tutorials." In these cases, a system that puts one great result at position 1 and garbage at positions 2-10 would score the same MRR as a system with great results at every position.

MAP (Mean Average Precision)

MAP extends beyond the first relevant result. For each query, it computes Average Precision (AP): the average of precision values calculated at each position where a relevant item appears. Then it averages AP across all queries.

Here's the key idea: AP rewards systems that cluster relevant results at the top. If positions 1, 2, and 3 are all relevant, precision at each of those positions is high, producing a high AP. If relevant results are scattered at positions 1, 5, and 9, the precision at positions 5 and 9 is diluted by the irrelevant results above them.

MAP treats relevance as binary: an item is either relevant or not. This works for information retrieval tasks with clear-cut relevance ("Does this document answer the query?") but struggles with recommendation systems where engagement is a spectrum, not a binary.

NDCG (Normalized Discounted Cumulative Gain)

NDCG is the standard ranking metric in production recommendation and search systems at companies like YouTube, Netflix, and Google. It solves two limitations of MAP: it handles graded relevance (not just relevant/irrelevant), and it applies a position-based discount (results further down the list contribute less).

The computation works in three steps.

Step 1: Assign relevance grades. Each item gets a relevance score. In a search system, this might be 3 (perfect match), 2 (good match), 1 (marginal), 0 (irrelevant). In a recommendation system, relevance might come from user engagement: watched the full video = 3, watched half = 2, clicked but bounced = 1, didn't click = 0.

Step 2: Compute DCG (Discounted Cumulative Gain). Sum the relevance scores, but discount by position using a logarithmic function: DCG = sum of (relevance_i / log2(i + 1)) for each position i. The log discount means position 1 gets full credit, position 2 gets 63% credit, position 5 gets 39% credit, and position 10 gets 30% credit.

Step 3: Normalize. Divide DCG by the ideal DCG (IDCG), which is the DCG you'd get if the items were sorted by relevance perfectly. NDCG = DCG / IDCG, giving a score between 0 and 1.

A concrete example with 5 items:

The NDCG of 0.88 reflects that the ranking is good but not perfect: the highly relevant Item E is stuck at position 5 instead of position 2, costing significant discounted gain.

Notice how the three metrics tell different stories about the same list. Precision@5 says 80% of results are relevant, which sounds great. MRR says the first relevant result is at position 1, which is perfect. But NDCG reveals the flaw: a highly relevant item (Item E, relevance 3) is buried at position 5 while an irrelevant item (Item B) sits at position 2. Only NDCG captures that ordering problem.

Picking a metric is a design decision with real consequences. A team that optimizes for the wrong metric builds a model that solves the wrong problem. The starting point is always the business objective, not the metric itself.

Fraud detection wants to catch as much fraud as possible while keeping false alarms manageable. The primary metric is typically recall at a fixed precision threshold: "What's our recall when precision is at least 95%?" This frames the problem correctly. Optimizing accuracy on a dataset where 0.1% of transactions are fraudulent is meaningless.

Spam filtering inverts the priority. A false positive (legitimate email marked as spam) is worse than a false negative (spam in the inbox) because users lose important messages. Precision matters more than recall. The primary metric might be precision at a fixed recall level, or simply precision@k for the top-scoring items.

ETA prediction has a direct user experience translation. MAE in minutes is the natural metric because it maps to what users feel: "the app said 12 minutes, the ride took 15, so the error was 3 minutes." MSE or RMSE would be appropriate if the business determines that large errors (being 20 minutes late) are disproportionately worse than small ones (being 2 minutes early). For a ride-hailing app, that's often the case.

Search ranking needs NDCG because both position and relevance degree matter. A search result that partially answers the query (relevance = 2) at position 1 is more valuable than a perfect answer (relevance = 3) at position 8. MAP would work if relevance were binary, but in practice, search relevance is graded.

Ad click prediction is unusual because it needs two things simultaneously: good ranking (show the most likely-to-click ads first) and good calibration (predicted probabilities must be accurate, not just correctly ordered). Ranking determines which ads to show. Calibration determines how much to bid. A model with AUC-ROC of 0.85 and well-calibrated probabilities can outperform a model with AUC-ROC of 0.90 and poor calibration because the bidding system relies on accurate probability estimates.

The Offline-Online Gap

One important caveat: offline metrics are proxies. A model with higher NDCG on a test set doesn't guarantee higher engagement in production. Netflix has documented cases where offline metric improvements didn't translate to measurable gains in user satisfaction. Spotify found similar disconnects between offline ranking metrics and actual listening behavior.

The gap exists because offline evaluation uses static historical data, while production behavior is dynamic: users react to what you show them, preferences shift, and the feedback loop between recommendations and user behavior changes the distribution. The next chapter dives deep into offline evaluation methodology and how to minimize this gap.

The Proxy Metric Problem

Even online metrics are proxies. Click-through rate measures clicks, not satisfaction. A clickbait headline gets high CTR but low user value. Watch time measures engagement, not learning. A confusing video that users rewatch to understand gets high watch time for the wrong reason.

The evaluation metric should be as close to the actual business objective as possible, but direct measurement of "user satisfaction" or "long-term value" is often impractical in an offline setting. This tension between measurable proxies and true objectives is a recurring theme in ML system design, and one that interviewers frequently probe.

In practice, production teams track multiple metrics, and model changes routinely improve some while worsening others. A new ranking model might improve NDCG by 2% but reduce result diversity by 5%. A fraud model update might increase recall by 3% but drop precision by 1.5%. These tradeoffs require a framework for decision-making, not just metric collection.

Primary and Guardrail Metrics

The most common framework in production is separating metrics into two categories: a primary metric that drives launch decisions and guardrail metrics that define acceptable boundaries.

The primary metric is what you're trying to improve. For a search ranking team, it might be NDCG@10. For a fraud team, recall at precision >= 0.95. For a recommendation team, user engagement measured by watch time.

Guardrail metrics are things that must not get worse. A search ranking change that improves NDCG but increases query latency beyond the p99 budget gets blocked. A recommendation change that improves engagement but reduces content diversity below a threshold gets blocked. The guardrails protect against tunnel-vision optimization.

Google's search quality team reportedly evaluates ranking changes against dozens of metrics, but launch decisions come down to: does NDCG improve, and do the guardrail metrics (latency, freshness, diversity, fairness) stay within bounds? This gives teams a clear decision rule instead of staring at a dashboard of 20 numbers trying to decide if the change is "net positive."

Resolution Strategies

When the primary and guardrail framework isn't enough, or when two primary metrics conflict, there are several approaches to resolving disagreements.

Weighted composite metrics (e.g., 0.7 NDCG + 0.3 diversity) produce a single number, which simplifies comparison. The problem is choosing the weights. What does it mean for 1% of NDCG to be worth 2.3% of diversity? These weights encode a business tradeoff that's rarely made explicit. And when the composite score improves, you can't tell whether both components improved or one improved enough to mask a regression in the other.

Constrained optimization is more principled: "maximize recall subject to precision >= 0.95." The constraint makes the tradeoff explicit and the threshold is usually grounded in a business requirement (below 95% precision, the manual review team can't keep up). The downside is that the threshold acts as a cliff: going from 95.1% to 95.0% precision is fine, but going from 95.0% to 94.9% blocks the launch entirely, even if the recall gain is enormous.

Pareto analysis is useful during research and model development. By mapping out all the model configurations where improving one metric necessarily worsens another, you identify the efficient frontier. The business then decides where on the frontier to operate. This works well when you're exploring a new problem space, but it doesn't scale to weekly launch decisions.

In practice, the primary + guardrail approach dominates. It's simple, it's explainable to stakeholders, and it forces teams to articulate what they're optimizing for and what they refuse to sacrifice.