{"title":"Overfitting vs Underfitting","description":"","content":"Diagnosing overfitting or underfitting is the first step, but it doesn’t solve the problem.\n\nThis chapter focuses on what to do next. It covers the practical tools: techniques to reduce overfitting, ways to address underfitting, and a simple framework to decide which approach to use in each case.\n\n---\n\n# Overfitting in Practice\n\nOverfitting is often described through the lens of bias and variance. In production, it has a more specific shape: the model performs well on historical data but degrades on new data in ways that are hard to predict.\n\nA fraud detection team at a fintech company trains a gradient-boosted tree on six months of labeled transactions. Offline evaluation looks strong: 0.96 AUC on a held-out test set. They deploy it. Within three weeks, precision drops from 92% to 74%. What happened? The model learned patterns that were artifacts of the training period rather than durable fraud signals. During those six months, a specific fraud ring was active, generating transactions with distinctive patterns (small amounts, late-night timing, specific merchant categories). The model latched onto those patterns. When that fraud ring was shut down and new fraud patterns emerged, the model's \"knowledge\" became stale. It was confidently wrong about the new patterns while still flagging phantom signals from the old ones.\n\nThis is overfitting's most dangerous form in production: the model doesn't just degrade, it degrades silently. Training metrics looked great. Test metrics looked great. The failure only surfaced when the real-world distribution shifted away from the training data.\n\n### Learning Curves: The Diagnostic Tool\n\nLearning curves plot model performance against training set size, and they're one of the clearest ways to see overfitting and underfitting in action.\n\n\n```mermaid\nflowchart LR\n subgraph OF[\"Overfitting\"]\n A[\"Training error:
stays very low\"]:::green\n B[\"Validation error:
high, doesn't converge
toward training\"]:::red\n C[\"Gap between curves
remains large\"]:::orange\n end\n\n subgraph UF[\"Underfitting\"]\n D[\"Training error:
stays high\"]:::red\n E[\"Validation error:
also high\"]:::red\n F[\"Both curves plateau
at poor performance\"]:::orange\n end\n\n subgraph GF[\"Good Fit\"]\n G[\"Training error:
low\"]:::green\n H[\"Validation error:
low, converges
toward training\"]:::green\n I[\"Small gap between
curves\"]:::teal\n end\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\nWith overfitting, the training error drops low while the validation error stays high. The gap between the two curves persists even as you add more data. With underfitting, both curves plateau at a high error, they're close together but both bad. A good fit shows both curves converging at a low error as training data increases.\n\nThe practical value of learning curves is that they tell you whether more data will help. If the validation curve is still trending downward as data increases, collecting more training data is worth the investment. If both curves have plateaued, more data won't fix the problem, you need a different model or better features.\n\n\n| Signal | Overfitting | Underfitting |\n|---|---|---|\n| Training error | Low | High |\n| Validation error | High (gap with training) | High (close to training) |\n| More data helps? | Yes, narrows the gap | No, both curves plateau |\n| More features help? | Usually makes it worse | Often helps |\n| More model complexity helps? | Makes it worse | Usually helps |\n| Production behavior | Degrades on new patterns | Consistently mediocre |\n\n\n---\n\n# Underfitting in Practice\n\nUnderfitting gets less attention than overfitting, but it's just as common in production systems, especially early in a project's lifecycle when teams default to simple models.\n\nA ride-sharing company builds an ETA prediction model using linear regression with three features: straight-line distance, time of day, and day of week. The model predicts that a 10-mile trip takes 25 minutes regardless of whether the route goes through downtown or along an empty highway. Training RMSE is 8.2 minutes. Validation RMSE is 8.5 minutes. Both numbers are bad, but they're close to each other. The model isn't memorizing the training data, it simply can't represent the relationship between route characteristics and travel time with a linear function.\n\nThe fix isn't regularization or dropout. The model needs more capacity: either a more complex model (gradient-boosted trees that can capture nonlinear interactions) or more informative features (traffic density, number of turns, road type). Adding L2 regularization to an underfitting model would make it worse by further constraining the already-limited capacity.\n\nThis distinction matters in system design interviews. When a candidate immediately jumps to \"add regularization\" without first checking whether the problem is overfitting or underfitting, it signals a mechanical rather than diagnostic approach.\n\n---\n\n# L1 and L2 Regularization\n\nRegularization adds a penalty term to the loss function that discourages the model from relying too heavily on any particular parameter. The core idea: a model with smaller, more evenly distributed weights is less likely to have memorized noise in the training data.\n\nWithout regularization, the model minimizes just the prediction error:\n\n**Loss = Prediction Error**\n\nWith regularization, the model minimizes prediction error plus a penalty on the weights:\n\n**Loss = Prediction Error + Penalty(weights)**\n\nThe two most common penalties work differently and produce different effects.\n\n### L2 Regularization (Ridge)\n\nL2 adds the sum of squared weights to the loss function. This pushes all weights toward zero but rarely makes any weight exactly zero. The effect is that the model spreads its \"attention\" across many features rather than relying heavily on a few.\n\nConsider a click prediction model with 200 features. Without regularization, the model might assign a weight of 15.3 to \"time since last click\" and near-zero weights to everything else. With L2 regularization, the model distributes importance more evenly across correlated features. If \"time since last click\" and \"session duration\" are correlated, both get moderate weights instead of one getting everything.\n\nL2 is the default choice for most production ML systems. It stabilizes training, reduces sensitivity to individual features, and rarely hurts performance.\n\n### L1 Regularization (Lasso)\n\nL1 adds the sum of absolute weights to the loss function. The mathematical difference from L2 is subtle, but the practical effect is dramatic: L1 drives some weights to exactly zero, effectively performing feature selection during training.\n\nThis is valuable when you suspect that many features are irrelevant. A model with 500 features might have only 30 that actually carry signal. L1 regularization identifies and zeros out the other 470, producing a sparse model that's cheaper to serve and easier to interpret.\n\n\n```python\nfrom sklearn.linear_model import LogisticRegression\n\n# L2 regularization (default) - shrinks all weights\nmodel_l2 = LogisticRegression(penalty='l2', C=1.0)\nmodel_l2.fit(X_train, y_train)\n\n# L1 regularization - zeros out irrelevant features\nmodel_l1 = LogisticRegression(penalty='l1', C=1.0, solver='liblinear')\nmodel_l1.fit(X_train, y_train)\n\n# Count non-zero features\nl2_features = sum(abs(w) > 1e-6 for w in model_l2.coef_[0])\nl1_features = sum(abs(w) > 1e-6 for w in model_l1.coef_[0])\n```\n\n\n**Output:**\n\n\n```plaintext\nL2: 200/200 features with non-zero weights\nL1: 34/200 features with non-zero weights\n```\n\n\nThe `C` parameter controls regularization strength. Smaller `C` means stronger regularization (more penalty on weights). This is one of the most commonly tuned hyperparameters in production ML.\n\n\n| Property | L1 (Lasso) | L2 (Ridge) |\n|---|---|---|\n| Effect on weights | Drives some to exactly zero | Shrinks all toward zero |\n| Feature selection | Yes, built-in | No |\n| Best when | Many irrelevant features | Many correlated features |\n| Model interpretability | Higher (fewer active features) | Lower (all features active) |\n| Serving cost | Lower (sparse model) | Higher (dense model) |\n| Typical use case | High-dimensional tabular data | Neural networks, dense features |\n| Risk | May drop useful features | May keep noisy features |\n\n\n### Regularization in Neural Networks (Weight Decay)\n\nIn deep learning, L2 regularization is typically called weight decay. The effect is the same: penalizing large weights to prevent the network from memorizing training data. PyTorch and TensorFlow both support weight decay as a parameter in the optimizer.\n\n\n```python\nimport torch.optim as optim\n\n# Adam optimizer with weight decay (L2 regularization)\noptimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)\n```\n\n\nTypical weight decay values range from 1e-4 to 1e-1. Too little and it has no effect. Too much and the model underfits because the penalty overwhelms the learning signal.\n\n---\n\n# Dropout\n\nDropout takes a different approach to regularization. Instead of penalizing weights, it randomly \"turns off\" a fraction of neurons during each training step. Each forward pass through the network uses a different random subset of neurons.\n\n\n```mermaid\nflowchart LR\n subgraph TR[\"Training: Dropout Active\"]\n I1[Input 1]:::primary --> H1[Neuron 1]:::primary\n I1 --> H2[Neuron 2
DROPPED]:::red\n I1 --> H3[Neuron 3]:::primary\n I2[Input 2]:::primary --> H1\n I2 --> H2\n I2 --> H3\n H1 --> O1[Output]:::green\n H3 --> O1\n end\n\n subgraph INF[\"Inference: All Active\"]\n I3[Input 1]:::primary --> H4[Neuron 1]:::teal\n I3 --> H5[Neuron 2]:::teal\n I3 --> H6[Neuron 3]:::teal\n I4[Input 2]:::primary --> H4\n I4 --> H5\n I4 --> H6\n H4 --> O2[Output]:::green\n H5 --> O2\n H6 --> O2\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\nDuring training, each neuron has a probability p of being dropped (set to zero) on any given forward pass. During inference, all neurons are active but their outputs are scaled to compensate for the fact that more neurons are contributing than during any single training step.\n\nWhy does this work? Without dropout, neurons can co-adapt: neuron A learns to rely on a very specific output from neuron B, and together they memorize a training pattern. Dropout breaks this co-adaptation. Since any neuron might be absent on any training step, each neuron is forced to learn features that are useful on their own, not just in combination with specific other neurons. The result is redundant, distributed representations that generalize better.\n\nDropout rates between 0.2 and 0.5 work well for most architectures. Lower rates (0.1-0.2) for input layers and embedding layers, higher rates (0.3-0.5) for large hidden layers. Applying too much dropout is equivalent to reducing model capacity, which can cause underfitting.\n\nOne important detail: dropout is only active during training. At inference time, all neurons are active. Forgetting to switch this off is a common bug. In PyTorch, `model.eval()` disables dropout (and batch normalization). Deploying a model in training mode produces noisy, non-deterministic predictions.\n\nDropout applies primarily to neural networks. For tree-based models like gradient-boosted trees, the analogous technique is subsampling: training each tree on a random subset of features and rows, which XGBoost and LightGBM support natively through `colsample_bytree` and `subsample` parameters.\n\n---\n\n# Early Stopping\n\nEarly stopping is the simplest and most universally applicable regularization technique. The idea: monitor validation loss during training and stop when it starts rising, even if training loss is still decreasing.\n\nThe logic is straightforward. At some point during training, the model transitions from learning real patterns to memorizing noise. Before that transition, both training and validation loss decrease. After it, training loss continues to decrease (the model is getting better at memorizing the training data) while validation loss increases (the model is getting worse at generalizing). The point where validation loss bottoms out is the optimal stopping point.\n\n\n```mermaid\nflowchart LR\n A[\"Epoch 1-20
Both losses
decrease\"]:::green\n B[\"Epoch 20-35
Val loss
plateaus\"]:::orange\n C[\"Epoch 35+
Val loss rises
Training loss
still drops\"]:::red\n D[\"STOP HERE
Best checkpoint
at epoch ~30\"]:::teal\n\n A --> B --> C\n B --> D\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n```\n\n\nIn practice, validation loss doesn't rise smoothly. It fluctuates. The \"patience\" parameter handles this: it specifies how many epochs of no improvement to tolerate before stopping. A patience of 5-10 epochs is typical. Too little patience and you stop too early during a temporary fluctuation. Too much patience and you waste compute training a model that's already overfitting.\n\nThe best practice is to save a checkpoint of the model weights every time validation loss reaches a new minimum. When training stops, you load the best checkpoint rather than the final weights. This way, even if you overshoot the optimal point by a few epochs, you still have the best model.\n\n\n```python\nbest_val_loss = float('inf')\npatience_counter = 0\npatience = 10\n\nfor epoch in range(max_epochs):\n train_loss = train_one_epoch(model, train_loader, optimizer)\n val_loss = evaluate(model, val_loader)\n\n if val_loss < best_val_loss:\n best_val_loss = val_loss\n patience_counter = 0\n torch.save(model.state_dict(), 'best_model.pt')\n else:\n patience_counter += 1\n\n if patience_counter >= patience:\n print(f\"Early stopping at epoch {epoch}\")\n break\n\n# Load best checkpoint for deployment\nmodel.load_state_dict(torch.load('best_model.pt'))\n```\n\n\nEarly stopping is almost always worth using. It costs nothing in model quality (you're just picking the best point on the training curve), reduces training compute, and works regardless of model architecture. It's the first technique to add when overfitting is a concern.\n\n---\n\n# Data Augmentation\n\nEvery other technique in this chapter works by constraining the model. Data augmentation takes the opposite approach: it expands the training data by creating modified versions of existing examples.\n\nThe core insight is that many transformations preserve the label. A photo of a cat is still a cat if you flip it horizontally, adjust the brightness, or crop it slightly. By training on these variations, the model learns to be invariant to transformations that don't affect the meaning, which directly reduces overfitting to the specific pixel patterns in the original training images.\n\n\n| Domain | Augmentation Technique | What It Does | Example |\n|---|---|---|---|\n| Images | Random crop, flip, rotation | Spatial invariance | Flipped photo of a cat is still a cat |\n| Images | Color jitter, brightness | Lighting invariance | Darker photo of a dog is still a dog |\n| Text | Synonym replacement | Vocabulary invariance | \"great movie\" and \"excellent film\" share the same sentiment |\n| Text | Back-translation | Paraphrase generation | Translate to French, then back to English |\n| Tabular | SMOTE (oversampling) | Class balance | Synthesize minority-class examples for fraud detection |\n| Tabular | Feature noise injection | Robustness | Add small Gaussian noise to continuous features |\n| Audio | Time stretching, pitch shift | Temporal invariance | Slightly faster speech is still the same words |\n\n\nData augmentation is most impactful in image-based systems, where it's considered essential rather than optional. ImageNet-trained models routinely use random crops, horizontal flips, and color jitter during training. For text and tabular data, augmentation techniques exist but tend to be more domain-specific and less universally effective.\n\nOne caveat: augmentation must preserve the label. Rotating a handwritten \"6\" by 180 degrees turns it into a \"9\". Aggressively cropping a product image might remove the product entirely. The augmentation strategy needs to be validated against the specific task.\n\nData augmentation is particularly effective when the training set is small. A medical imaging team with 2,000 labeled X-rays can effectively multiply their dataset to 20,000 through augmentations, significantly reducing overfitting for a deep CNN that would otherwise memorize 2,000 images quickly.\n\n---\n\n# Choosing the Right Fix\n\nEach technique targets a different aspect of the problem. Using the right one depends on whether you're fighting overfitting or underfitting, what kind of model you're using, and how much data you have.\n\n\n```mermaid\nflowchart TD\n A[\"Model
Underperforming\"]:::primary --> B{\"Problem
Type?\"}:::orange\n\n B -->|\"Underfitting
(high train error)\"| C[\"Increase Capacity\"]:::teal\n B -->|\"Overfitting
(train-val gap)\"| D[\"Constrain Model\"]:::teal\n\n C --> C1[\"More complex model\"]:::green\n C --> C2[\"Add features\"]:::green\n C --> C3[\"Reduce regularization\"]:::green\n\n D --> D1[\"Early stopping
(always start here)\"]:::green\n D --> D2[\"L2 / weight decay\"]:::green\n D --> D3[\"Dropout
(neural nets)\"]:::green\n D --> D4[\"Data augmentation
(if applicable)\"]:::green\n D --> D5[\"More training data\"]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nFor overfitting, start with early stopping because it's free and effective. Then add L2 regularization or weight decay. If you're using a neural network, add dropout. If the domain supports it, add data augmentation. If you have the budget, collect more training data.\n\nFor underfitting, the fixes go in the opposite direction. Increase model complexity (more layers, more trees, wider networks). Add more informative features. If regularization is already applied, reduce it. Don't add more regularization to an underfitting model, it will make the problem worse.\n\nIn practice, production models often combine multiple techniques. A typical neural network setup uses weight decay in the optimizer, dropout in the hidden layers, data augmentation in the data pipeline, and early stopping in the training loop. These techniques are complementary, not redundant: weight decay constrains individual weights, dropout prevents co-adaptation between neurons, augmentation expands the effective training distribution, and early stopping picks the optimal training duration.\n\n\n| Technique | Targets | Model Types | Compute Cost | When to Avoid |\n|---|---|---|---|---|\n| Early stopping | Overfitting | All | None (saves compute) | When training is already cheap |\n| L2 / weight decay | Overfitting | All | Negligible | When model is underfitting |\n| L1 | Overfitting + feature selection | Linear, tree-based | Negligible | When all features are relevant |\n| Dropout | Overfitting | Neural networks | Slightly longer training | Small networks, tree models |\n| Data augmentation | Overfitting | All (especially CNNs) | More training time, storage | When augmentations change labels |\n| More data | Overfitting | All | Collection cost | When underfitting (won't help) |\n| More features | Underfitting | All | Feature engineering time | When already overfitting |\n| More complexity | Underfitting | All | More compute, serving cost | When already overfitting |\n","pageType":"ml-system-design"}

Overfitting vs Underfitting

Ashish Pratap Singh

Get Premium