Diagnosing overfitting or underfitting is the first step, but it doesn’t solve the problem.

This chapter focuses on what to do next. It covers the practical tools: techniques to reduce overfitting, ways to address underfitting, and a simple framework to decide which approach to use in each case.

The previous chapter defined overfitting through the lens of bias and variance. In production, overfitting has a more specific shape: the model performs well on historical data but degrades on new data in ways that are hard to predict.

A fraud detection team at a fintech company trains a gradient-boosted tree on six months of labeled transactions. Offline evaluation looks strong: 0.96 AUC on a held-out test set. They deploy it. Within three weeks, precision drops from 92% to 74%. What happened? The model learned patterns that were artifacts of the training period rather than durable fraud signals. During those six months, a specific fraud ring was active, generating transactions with distinctive patterns (small amounts, late-night timing, specific merchant categories). The model latched onto those patterns. When that fraud ring was shut down and new fraud patterns emerged, the model's "knowledge" became stale. It was confidently wrong about the new patterns while still flagging phantom signals from the old ones.

This is overfitting's most dangerous form in production: the model doesn't just degrade, it degrades silently. Training metrics looked great. Test metrics looked great. The failure only surfaced when the real-world distribution shifted away from the training data.

Learning Curves: The Diagnostic Tool

Learning curves plot model performance against training set size, and they're one of the clearest ways to see overfitting and underfitting in action.

With overfitting, the training error drops low while the validation error stays high. The gap between the two curves persists even as you add more data. With underfitting, both curves plateau at a high error, they're close together but both bad. A good fit shows both curves converging at a low error as training data increases.

The practical value of learning curves is that they tell you whether more data will help. If the validation curve is still trending downward as data increases, collecting more training data is worth the investment. If both curves have plateaued, more data won't fix the problem, you need a different model or better features.

Underfitting gets less attention than overfitting, but it's just as common in production systems, especially early in a project's lifecycle when teams default to simple models.

A ride-sharing company builds an ETA prediction model using linear regression with three features: straight-line distance, time of day, and day of week. The model predicts that a 10-mile trip takes 25 minutes regardless of whether the route goes through downtown or along an empty highway. Training RMSE is 8.2 minutes. Validation RMSE is 8.5 minutes. Both numbers are bad, but they're close to each other. The model isn't memorizing the training data, it simply can't represent the relationship between route characteristics and travel time with a linear function.

The fix isn't regularization or dropout. The model needs more capacity: either a more complex model (gradient-boosted trees that can capture nonlinear interactions) or more informative features (traffic density, number of turns, road type). Adding L2 regularization to an underfitting model would make it worse by further constraining the already-limited capacity.

This distinction matters in system design interviews. When a candidate immediately jumps to "add regularization" without first checking whether the problem is overfitting or underfitting, it signals a mechanical rather than diagnostic approach.

Regularization adds a penalty term to the loss function that discourages the model from relying too heavily on any particular parameter. The core idea: a model with smaller, more evenly distributed weights is less likely to have memorized noise in the training data.

Without regularization, the model minimizes just the prediction error:

Loss = Prediction Error

With regularization, the model minimizes prediction error plus a penalty on the weights:

Loss = Prediction Error + Penalty(weights)

The two most common penalties work differently and produce different effects.

L2 Regularization (Ridge)

L2 adds the sum of squared weights to the loss function. This pushes all weights toward zero but rarely makes any weight exactly zero. The effect is that the model spreads its "attention" across many features rather than relying heavily on a few.

Consider a click prediction model with 200 features. Without regularization, the model might assign a weight of 15.3 to "time since last click" and near-zero weights to everything else. With L2 regularization, the model distributes importance more evenly across correlated features. If "time since last click" and "session duration" are correlated, both get moderate weights instead of one getting everything.

L2 is the default choice for most production ML systems. It stabilizes training, reduces sensitivity to individual features, and rarely hurts performance.

L1 Regularization (Lasso)

L1 adds the sum of absolute weights to the loss function. The mathematical difference from L2 is subtle, but the practical effect is dramatic: L1 drives some weights to exactly zero, effectively performing feature selection during training.

This is valuable when you suspect that many features are irrelevant. A model with 500 features might have only 30 that actually carry signal. L1 regularization identifies and zeros out the other 470, producing a sparse model that's cheaper to serve and easier to interpret.

Output:

The C parameter controls regularization strength. Smaller C means stronger regularization (more penalty on weights). This is one of the most commonly tuned hyperparameters in production ML.

Regularization in Neural Networks (Weight Decay)

In deep learning, L2 regularization is typically called weight decay. The effect is the same: penalizing large weights to prevent the network from memorizing training data. PyTorch and TensorFlow both support weight decay as a parameter in the optimizer.

Typical weight decay values range from 1e-4 to 1e-1. Too little and it has no effect. Too much and the model underfits because the penalty overwhelms the learning signal.

Dropout takes a different approach to regularization. Instead of penalizing weights, it randomly "turns off" a fraction of neurons during each training step. Each forward pass through the network uses a different random subset of neurons.

During training, each neuron has a probability p of being dropped (set to zero) on any given forward pass. During inference, all neurons are active but their outputs are scaled to compensate for the fact that more neurons are contributing than during any single training step.

Why does this work? Without dropout, neurons can co-adapt: neuron A learns to rely on a very specific output from neuron B, and together they memorize a training pattern. Dropout breaks this co-adaptation. Since any neuron might be absent on any training step, each neuron is forced to learn features that are useful on their own, not just in combination with specific other neurons. The result is redundant, distributed representations that generalize better.

Dropout rates between 0.2 and 0.5 work well for most architectures. Lower rates (0.1-0.2) for input layers and embedding layers, higher rates (0.3-0.5) for large hidden layers. Applying too much dropout is equivalent to reducing model capacity, which can cause underfitting.

One important detail: dropout is only active during training. At inference time, all neurons are active. Forgetting to switch this off is a common bug. In PyTorch, model.eval() disables dropout (and batch normalization). Deploying a model in training mode produces noisy, non-deterministic predictions.

Dropout applies primarily to neural networks. For tree-based models like gradient-boosted trees, the analogous technique is subsampling: training each tree on a random subset of features and rows, which XGBoost and LightGBM support natively through colsample_bytree and subsample parameters.

Early stopping is the simplest and most universally applicable regularization technique. The idea: monitor validation loss during training and stop when it starts rising, even if training loss is still decreasing.

The logic is straightforward. At some point during training, the model transitions from learning real patterns to memorizing noise. Before that transition, both training and validation loss decrease. After it, training loss continues to decrease (the model is getting better at memorizing the training data) while validation loss increases (the model is getting worse at generalizing). The point where validation loss bottoms out is the optimal stopping point.

In practice, validation loss doesn't rise smoothly. It fluctuates. The "patience" parameter handles this: it specifies how many epochs of no improvement to tolerate before stopping. A patience of 5-10 epochs is typical. Too little patience and you stop too early during a temporary fluctuation. Too much patience and you waste compute training a model that's already overfitting.

The best practice is to save a checkpoint of the model weights every time validation loss reaches a new minimum. When training stops, you load the best checkpoint rather than the final weights. This way, even if you overshoot the optimal point by a few epochs, you still have the best model.

Early stopping is almost always worth using. It costs nothing in model quality (you're just picking the best point on the training curve), reduces training compute, and works regardless of model architecture. It's the first technique to add when overfitting is a concern.

Every other technique in this chapter works by constraining the model. Data augmentation takes the opposite approach: it expands the training data by creating modified versions of existing examples.

The core insight is that many transformations preserve the label. A photo of a cat is still a cat if you flip it horizontally, adjust the brightness, or crop it slightly. By training on these variations, the model learns to be invariant to transformations that don't affect the meaning, which directly reduces overfitting to the specific pixel patterns in the original training images.

Data augmentation is most impactful in image-based systems, where it's considered essential rather than optional. ImageNet-trained models routinely use random crops, horizontal flips, and color jitter during training. For text and tabular data, augmentation techniques exist but tend to be more domain-specific and less universally effective.

One caveat: augmentation must preserve the label. Rotating a handwritten "6" by 180 degrees turns it into a "9". Aggressively cropping a product image might remove the product entirely. The augmentation strategy needs to be validated against the specific task.

Data augmentation is particularly effective when the training set is small. A medical imaging team with 2,000 labeled X-rays can effectively multiply their dataset to 20,000 through augmentations, significantly reducing overfitting for a deep CNN that would otherwise memorize 2,000 images quickly.

Each technique targets a different aspect of the problem. Using the right one depends on whether you're fighting overfitting or underfitting, what kind of model you're using, and how much data you have.

For overfitting, start with early stopping because it's free and effective. Then add L2 regularization or weight decay. If you're using a neural network, add dropout. If the domain supports it, add data augmentation. If you have the budget, collect more training data.

For underfitting, the fixes go in the opposite direction. Increase model complexity (more layers, more trees, wider networks). Add more informative features. If regularization is already applied, reduce it. Don't add more regularization to an underfitting model, it will make the problem worse.

In practice, production models often combine multiple techniques. A typical neural network setup uses weight decay in the optimizer, dropout in the hidden layers, data augmentation in the data pipeline, and early stopping in the training loop. These techniques are complementary, not redundant: weight decay constrains individual weights, dropout prevents co-adaptation between neurons, augmentation expands the effective training distribution, and early stopping picks the optimal training duration.