The previous chapter introduced classification and regression, the two core tasks in supervised learning. But identifying the task is only the starting point. It does not tell you which algorithm to choose.

Take fraud detection as an example. You could build it using logistic regression, a gradient-boosted tree, or a neural network. Each option comes with very different tradeoffs in training cost, serving latency, interpretability, and accuracy.

In practice, most production ML systems rely on a small set of algorithm families. Knowing when to use each one is one of the most practical and high-impact skills in ML system design.

Linear models are the oldest and simplest family, yet they remain one of the most widely used in production. The idea is straightforward: take a weighted sum of input features, optionally pass it through a sigmoid function for classification, and output a prediction.

Logistic regression applies a sigmoid to the weighted sum to produce a probability between 0 and 1. Linear regression skips the sigmoid and outputs the weighted sum directly as a continuous value. In both cases, the model learns by adjusting weights to minimize a loss function on training data.

So why do these simple models persist when more powerful alternatives exist? The reasons are practical.

Speed

Linear models train in minutes on billions of examples and serve predictions in microseconds. Google's early ad click-through rate prediction system ran on logistic regression at massive scale, handling billions of predictions per day.

When your system needs to score thousands of candidates per request with single-digit millisecond latency budgets, microsecond inference matters.

Interpretability

Each feature gets a weight, and you can directly read which features push predictions up or down. Credit scoring models at banks use logistic regression partly because regulators require explanations for why a loan was denied.

You can point to the specific feature weights: high debt-to-income ratio contributed -0.8, short credit history contributed -0.3. Try explaining a 200-tree gradient-boosted model to a regulator.

Stability

Small changes in training data produce small changes in predictions. There are no wild swings from random initialization or hyperparameter sensitivity. This predictability matters when models run in high-stakes production systems.

The fundamental limitation is expressiveness. Linear models can only learn linear decision boundaries. If the relationship between features and the target is nonlinear (and it usually is), the model underfits.

You can partially fix this by manually engineering nonlinear features: polynomial terms, interaction features, bucketed numericals. Google's ad system used logistic regression with billions of manually engineered feature crosses. But this pushes complexity from the model into the feature engineering pipeline, which brings its own maintenance burden.

This is the "feature engineering tax." Linear models are cheap to run but expensive to feed. Every nonlinear pattern you want the model to capture requires an engineer to identify it, build it, and maintain it. As the number of patterns grows, this approach doesn't scale.

Despite this limitation, linear models serve a critical role: they're the baseline every production team should start with. If a logistic regression achieves 90% of the performance you need with 10% of the complexity, that's often the right trade-off. You always want to know how much accuracy the fancier model actually buys you.

Tree-based models work fundamentally differently from linear models. Instead of learning a weighted sum, they learn a series of if-else decision rules that recursively split the data.

A single decision tree asks questions like "Is the transaction amount greater than $5,000?" and "Is the merchant category 'electronics'?" at each node, splitting the data until it reaches a leaf node with a prediction.

This is intuitive and highly interpretable. But single decision trees have a serious problem: they overfit badly. A deep tree memorizes the training data instead of learning generalizable patterns.

Two techniques fix this, and the difference between them matters.

Random forests train many trees in parallel, each on a random subset of the data and features, then average their predictions. This reduces overfitting through diversity: individual trees make different errors, and averaging cancels them out. Random forests are solid, reliable, and hard to misconfigure. But they're not state-of-the-art.

Gradient-boosted trees (GBTs) take a different approach. Instead of training trees independently, they train trees sequentially. The first tree makes predictions. The second tree focuses on the errors the first tree made. The third tree focuses on the remaining errors. Each successive tree corrects the mistakes of all previous trees. The final prediction is the sum of all trees' contributions.

The gradient boosting mechanism is worth understanding in more detail, since it's the algorithm behind XGBoost, LightGBM, and CatBoost, the three most popular implementations.

Each tree is typically shallow (depth 4-8), so it captures a simple pattern. But hundreds of these shallow trees stacked together can model complex nonlinear relationships. The learning rate controls how much each tree contributes, preventing any single tree from dominating.

GBTs dominate tabular data in production, and understanding why is important for making algorithm choices.

They handle mixed feature types natively

Tabular data has categorical columns (merchant category, device type), numerical columns (transaction amount, user age), and often missing values. GBTs handle all of these without preprocessing. You don't need to one-hot encode categoricals or impute missing values. The tree splits naturally partition both types.

They capture feature interactions automatically

A tree that splits on "transaction amount > $5,000" at the root and "merchant category = electronics" at the next level has learned an interaction: high-value electronics purchases behave differently. Linear models need these interactions engineered manually.

They train fast and serve fast

Training on a dataset with millions of rows and hundreds of features takes minutes to hours on CPU. Serving requires traversing a few hundred shallow trees, which completes in sub-millisecond time. No GPU needed.

Airbnb used XGBoost as the core of their search ranking system for years. Uber's ETA prediction runs on gradient-boosted trees with features like distance, traffic conditions, and time of day. Fraud detection systems at companies like Stripe and PayPal rely heavily on GBTs because the data is inherently tabular (transaction metadata) and the models need to serve in real-time.

In Kaggle competitions, GBTs win the vast majority of tabular data challenges. This isn't just an academic observation. It reflects a genuine performance advantage that carries over to production.

GBTs do have limitations. They extrapolate poorly: if the training data contains transaction amounts from $1 to $10,000, the model has no principled way to handle a $100,000 transaction. They also struggle with very high-dimensional sparse data (like raw text represented as bag-of-words with millions of vocabulary terms), where neural approaches have an advantage.

Neural networks stack layers of artificial neurons, where each neuron computes a weighted sum of its inputs, adds a bias, and passes the result through a nonlinear activation function. The power comes from depth: multiple layers can learn hierarchical representations, capturing increasingly abstract patterns.

Three architectures cover the major use cases in production ML.

Feedforward Networks (MLPs)

A multi-layer perceptron (MLP) is the simplest neural network: fully connected layers stacked on top of each other. Input features pass through hidden layers, each applying a learned transformation, and produce an output.

MLPs on their own aren't special for tabular data. They rarely outperform gradient-boosted trees. Where they shine is as a component in larger systems. YouTube's Wide & Deep architecture combines a linear model (the "wide" path for memorization) with an MLP (the "deep" path for generalization).

The MLP takes dense feature vectors, including pre-computed embeddings, and learns complex interaction patterns that the linear component misses. This architecture and its variants power ranking systems across Google's products.

CNNs for Images

Convolutional neural networks exploit a key property of images: spatial locality. A pixel's meaning depends on its neighbors. Convolutional filters slide across the image, detecting patterns like edges, textures, and shapes at progressively higher abstraction levels. Early layers detect edges. Middle layers detect parts (eyes, wheels). Deep layers detect objects.

This hierarchical feature learning is what makes CNNs transformative. Before CNNs, image classification required hand-engineered features (color histograms, edge detectors, texture descriptors). An engineer had to decide which visual patterns mattered. CNNs learn these features directly from raw pixels, and they learn better features than humans design.

Google Photos uses CNNs for image search and organization. Pinterest's visual search lets users find products by photographing them, powered by CNN-based visual similarity. Content moderation systems at every major platform use CNNs to detect policy violations in uploaded images and video frames.

RNNs and LSTMs for Sequences

Recurrent neural networks process data one step at a time, maintaining a hidden state that carries information from previous steps. This makes them a natural fit for sequential data: text (word by word), time series (timestep by timestep), and user event sequences (action by action).

Standard RNNs struggle with long sequences because gradients either vanish (becoming too small to learn from) or explode (becoming too large and destabilizing training) as they propagate through many timesteps. LSTMs (Long Short-Term Memory networks) fix this with a gating mechanism that controls what information to keep, forget, or output at each step.

Before transformers took over NLP, LSTMs were the dominant architecture for text processing. They still appear in production systems for time-series forecasting and session-based recommendation, where the sequential nature of user behavior is the primary signal. Netflix has used recurrent architectures to model viewing sessions, predicting what a user wants to watch next based on their recent activity.

Transformers have largely replaced RNNs for text and are expanding into vision and audio. They process all positions in parallel (rather than sequentially) and use attention mechanisms to capture long-range dependencies. This chapter won't deep-dive into transformers, but they represent the current frontier for sequence modeling and increasingly for other data types.

The common thread across all neural architectures: they learn feature representations from raw data. You don't need to manually engineer "is there a face in this image" or "does this sentence express negative sentiment." The network discovers these patterns on its own. The trade-off is that this learning requires large datasets (often hundreds of thousands to millions of examples), expensive compute (GPU training for hours or days), and the resulting models are difficult to interpret and debug.

Choosing an algorithm isn't about which one is "best" in the abstract. It's about which one fits the constraints of your specific system.

The single strongest signal is data type. This one factor drives the decision more than anything else.

For tabular data (structured rows and columns: user attributes, transaction metadata, sensor readings), the path is well-established. Start with logistic regression or linear regression as a baseline. If you need better accuracy, move to gradient-boosted trees. XGBoost or LightGBM will almost always outperform the linear baseline, often significantly.

Neural networks for tabular data rarely outperform well-tuned GBTs, and they're harder to train, serve, and debug. Research papers occasionally show neural networks matching GBTs on tabular benchmarks, but in practice the operational overhead isn't worth marginal gains.

For unstructured data (images, text, audio, video), neural networks are the only practical choice. You can't hand-engineer features from raw pixels or text tokens at the scale production systems require. A CNN will extract visual features that no amount of manual feature engineering could match. A transformer will capture language patterns that bag-of-words models miss entirely.

Beyond data type, four secondary factors shape the decision:

Interpretability

Regulated industries (banking, healthcare, insurance) often require model explanations. Linear models provide inherent interpretability through feature weights. GBTs can be explained post-hoc using tools like SHAP, but the explanations are approximations. Neural networks are the hardest to interpret. If a regulator asks "why was this loan denied?", your answer needs to be specific and defensible.

Training budget

Linear models train in minutes on CPU. GBTs train in minutes to hours on CPU. Neural networks train in hours to days on GPU. If you're iterating quickly on a new problem and need to test dozens of feature combinations, the fast feedback loop of linear models or GBTs is valuable. If you have a mature system where 0.1% accuracy improvement justifies GPU costs, neural networks become viable.

Serving latency

Linear models serve in microseconds. GBTs serve in sub-millisecond time. Small MLPs take 1-5ms. CNNs and transformers need 10-100ms and often require GPU inference. A system with a 5ms total latency budget can't afford a transformer in the serving path unless it pre-computes predictions in batch.

Data volume

Linear models can learn useful patterns from thousands of examples. GBTs need tens of thousands. Neural networks typically need hundreds of thousands to millions before they outperform simpler alternatives. With a small dataset, a complex model just memorizes the training data.

One principle ties this together: start simple and upgrade when measured improvements justify added complexity. Every production team should know how well a logistic regression or simple GBT performs on their problem before reaching for neural networks. If the simple model gets you 90% of the way, the remaining 10% may not be worth the operational burden of training, serving, and debugging a deep network.

The split between "GBTs for tabular, neural nets for unstructured" isn't arbitrary. It reflects a fundamental difference in how these data types are structured.

Tabular data comes with pre-defined, semantically meaningful columns. "Transaction amount," "merchant category," "user age" are already meaningful features. Tree-based models exploit this structure directly. A decision tree split on "transaction amount > $5,000" is a natural, axis-aligned partition that matches how the data is organized. Trees are essentially learning a set of rules over pre-defined features, and that's exactly what tabular data provides.

Neural networks don't have this structural advantage on tabular data. Their gradient-based optimization has to learn which feature boundaries matter from scratch, without the inductive bias that axis-aligned splits provide.

The heterogeneous nature of tabular features (some categorical, some numerical, some ordinal, some with missing values) also creates challenges for the uniform tensor operations that neural networks rely on.

Unstructured data is the opposite situation. Raw pixels, text tokens, and audio waveforms carry no pre-defined semantic meaning. Pixel 1,247 in an image means nothing on its own. The meaning emerges from spatial patterns across thousands of pixels. No human can manually specify the features that distinguish a cat from a dog at production quality.

Neural networks learn these hierarchical representations automatically: edges combine into textures, textures into parts, parts into objects. Trees fundamentally can't do this because a decision tree can't split on "this region of the image contains a face."

In practice, production systems often combine both approaches. A recommendation system might use a neural network to generate embeddings from item images and text descriptions (extracting features from unstructured data), then feed those embeddings as input features into a gradient-boosted tree alongside tabular features like item popularity, user tenure, and time of day.

This hybrid pattern is common at companies operating at scale.