The previous chapter mentioned that neural networks can produce something more valuable than predictions: embeddings. These dense vector representations have become one of the most versatile building blocks in production ML.

Recommendation systems, search engines, fraud detection, ad ranking, nearly every ML system at scale relies on embeddings somewhere in its pipeline. Understanding what they are, how they're learned, and how they're used in production is essential for ML system design.

Before embeddings existed, ML systems represented categorical data using one-hot encoding. Each category gets its own binary column: a 1 in the column for that category, 0 everywhere else.

Consider a movie recommendation system with 50,000 movies. To represent "The Dark Knight," you'd create a vector of 50,000 dimensions with a single 1 at position 8,432 and zeros everywhere else. Now add 10 million users, each also one-hot encoded. The feature space is enormous, almost entirely zeros, and completely wasteful.

But the bigger problem isn't size. It's that one-hot encoding captures no relationships between entities. The vectors for "The Dark Knight" and "Batman Begins" are just as different as the vectors for "The Dark Knight" and "Finding Nemo." Orthogonal vectors in every case, with zero overlap. The encoding treats every movie as equally unrelated to every other movie, which is obviously wrong.

This creates three concrete problems for ML systems:

One-hot encoding forces the model to learn every relationship from scratch. With embeddings, a model that learned "users who like The Dark Knight also like Inception" can generalize to Batman Begins without ever seeing that specific pair in training data, because those movies sit close together in embedding space.

An embedding is a dense vector of floating-point numbers that represents an entity (a word, a user, a movie, a product) in a continuous vector space. The key property: entities that are semantically similar end up close together in this space.

Each dimension in an embedding captures some latent property. For movie embeddings, one dimension might roughly correspond to "how much action is in this film," another to "how dark the tone is," another to "target audience age."

These dimensions aren't hand-designed. They emerge automatically during training, and they usually don't map to any single human-interpretable concept. But the geometry they create is meaningful.

In this simplified 2D view, Nolan's dark, complex films cluster together, animated family movies cluster separately, and the distance between clusters reflects how different those categories are.

The most famous demonstration of embedding properties comes from word embeddings. Google's Word2vec showed that vector arithmetic on word embeddings captures semantic relationships:

These relationships weren't programmed. They emerged from training on large text corpora, because words that appear in similar contexts develop similar vectors. "King" and "queen" appear in similar sentences, differing primarily along a gender axis, so their vectors differ by roughly the same offset as "man" and "woman."

This property, that geometric operations in embedding space correspond to semantic operations, is what makes embeddings so powerful. They turn unstructured similarity ("these movies feel similar") into structured math (vector distance computation).

Embeddings aren't handcrafted. They start as random vectors and are refined through training on data that contains implicit or explicit signals about what should be similar to what. The training process adjusts vectors so that related entities end up close together and unrelated entities end up far apart.

Word2vec: The Breakthrough Idea

Word2vec, published by Google in 2013, was the paper that made embeddings mainstream. The core insight: you can learn meaningful word representations from a simple prediction task.

The Skip-gram variant works by predicting surrounding words from a center word. Given the sentence "the cat sat on the mat," with "sat" as the center word and a window size of 2, the model tries to predict "the," "cat," "on," and "the" from the input "sat."

The embedding layer is a matrix of size vocabulary_size x embedding_dim. Each row is one word's embedding vector. During training, the model adjusts these vectors to make the prediction task easier. Words that frequently appear in similar contexts ("dog" and "cat" both appear near "pet," "feed," "vet") develop similar vectors because similar vectors make the same context predictions.

The model itself is shallow, just one hidden layer. The embeddings aren't the output. They're the learned weights of the hidden layer. The prediction task is just a means to an end: a pretext task that forces the model to learn useful representations.

This is the fundamental pattern behind all embedding learning: design a task where good embeddings make the task easier, then train a model on that task and extract the embeddings.

Item2vec and Collaborative Filtering Embeddings

The same idea generalizes far beyond words. Replace "words in a sentence" with "items in a user session," and you get item embeddings.

If a user watches "The Dark Knight," then "Inception," then "Interstellar" in one session, those movies are "context" for each other, just like words in a sentence. Training a Skip-gram model on viewing sessions produces movie embeddings where films frequently watched together end up close in the vector space.

Spotify uses a version of this approach called Word2vec on playlists. Songs that appear in the same playlists frequently develop similar embeddings. This powers their Discover Weekly recommendations: find songs close to what you've listened to but that you haven't heard yet.

YouTube applies a similar technique at massive scale. Their candidate generation model treats video watch sequences as "sentences" and learns video embeddings. A user's watch history gets averaged into a user embedding, and recommendations come from finding videos whose embeddings are close to that user vector.

Collaborative filtering embeddings take a slightly different approach. Instead of treating items as sequences, they factorize the user-item interaction matrix. The idea: if you have a matrix where rows are users, columns are items, and cells are ratings (or implicit signals like clicks), you can decompose it into two lower-dimensional matrices.

One gives you user embeddings, the other gives you item embeddings. A user's predicted rating for an item is the dot product of their respective embeddings.

Netflix's early recommendation system used matrix factorization heavily. The Netflix Prize competition demonstrated that this approach, despite its simplicity, captures nuanced user preferences. A user who rates both "The Godfather" and "Goodfellas" highly develops an embedding that sits in the "crime drama" region of the space, making other films in that region natural recommendations.

Pre-trained Embeddings

Training embeddings from scratch requires substantial data. If you're building a text classification system with 10,000 labeled examples, you don't have enough data to learn good word embeddings. But someone else already trained embeddings on billions of words.

Pre-trained embeddings transfer knowledge from large-scale training to smaller downstream tasks. BERT produces contextual word embeddings from training on massive text corpora. ResNet produces image embeddings from training on millions of labeled images. These embeddings capture general semantic knowledge that transfers to specific tasks.

The decision of when to use pre-trained versus custom embeddings depends on data volume and domain specificity:

In practice, production systems often use both. An e-commerce recommendation system might use pre-trained BERT embeddings for product descriptions (text understanding), pre-trained ResNet embeddings for product images (visual similarity), and custom-trained user/product embeddings for collaborative filtering (behavioral similarity). These different embedding types capture different facets of similarity and get combined in downstream models.

Embedding tables are arguably the most common component in production ML systems. Every recommendation system, search ranking model, and ad click prediction model at scale uses them.

An embedding table is a lookup structure: given an entity ID (user ID, item ID, ad ID), return its embedding vector. Conceptually, it's a dictionary mapping IDs to fixed-size vectors. At serving time, the model receives a user ID and a list of candidate item IDs, looks up their embeddings, and uses those vectors as input features.

The scale of these tables is significant. A system serving 100 million users with 128-dimensional embeddings stored as 32-bit floats needs roughly 48 GB just for user embeddings. Add item embeddings, ad embeddings, and query embeddings, and the total memory footprint grows quickly.

This memory pressure drives several engineering decisions. Large embedding tables get sharded across multiple machines, with each machine holding a subset of the rows.

Companies like Meta and Google built custom parameter servers specifically for serving embedding lookups at low latency across distributed tables. Some systems use mixed-precision storage (float16 instead of float32) to halve memory at minimal quality loss.

Embedding tables also need regular updates. User preferences change, new products launch, item popularity shifts. Systems typically retrain embeddings on a schedule (daily or weekly) and push updated tables to the serving infrastructure.

The update mechanism matters: you need to swap tables atomically without dropping serving traffic, and new embeddings should be compatible with the existing model.

Once entities are embedded in a shared vector space, finding similar entities becomes a distance computation. This enables a fundamental operation in ML systems: given a query entity, find the most similar entities from a large catalog.

Distance Metrics

Three distance metrics dominate embedding-based similarity:

Cosine similarity is the most commonly used in practice. It measures the angle between two vectors, ignoring their magnitudes. This is useful because embedding magnitudes can vary for reasons unrelated to similarity (training artifacts, frequency effects). Two movie embeddings pointing in similar directions are semantically similar regardless of their vector lengths.

Dot product is preferred when magnitude carries meaning. In recommendation systems, popular items often have larger embedding norms.

Using dot product instead of cosine similarity naturally boosts popular items, which can be desirable. YouTube's candidate generation model uses dot product for this reason.

Approximate Nearest Neighbors

Brute-force similarity search computes the distance between the query and every entity in the catalog. With 10 million items and 128-dimensional embeddings, that's 10 million dot products per query.

At 100 queries per second, this costs roughly 1.28 billion floating-point operations per second, and latency grows linearly with catalog size. For systems with hundreds of millions of items, brute force is too slow.

Approximate Nearest Neighbor (ANN) algorithms trade a small amount of accuracy for dramatic speed improvements. Instead of checking every item, they use index structures to narrow the search to a small subset of candidates likely to contain the true nearest neighbors.

The major ANN algorithms each use different strategies to organize the embedding space:

HNSW (Hierarchical Navigable Small World) builds a multi-layer graph where each node connects to its approximate nearest neighbors.

Searching starts at the top layer (sparse, long-range connections) and drills down to lower layers (dense, local connections). It's the default choice for many production systems because it offers high recall with low latency, at the cost of higher memory usage.

IVF (Inverted File Index) clusters the embedding space using k-means, then at query time only searches the nearest clusters. The trade-off is controlled by the number of clusters probed: more clusters means higher recall but slower search.

Product Quantization compresses embeddings by splitting each vector into sub-vectors and quantizing each sub-vector to its nearest centroid from a learned codebook. This dramatically reduces memory (a 128-dim float32 vector can be compressed from 512 bytes to 16 bytes) while still enabling fast approximate distance computation.

In practice, systems often combine these approaches. Meta's Faiss library implements IVF+PQ (cluster-based search with compressed vectors) for their recommendation system, handling billions of embeddings with sub-millisecond search latency.

Embeddings are powerful but introduce their own failure modes. Understanding these is critical for system design because they affect architecture decisions.

Cold Start

New entities have no embedding. A new user who just signed up has no interaction history, so there's no data to learn their embedding from. A new product added to the catalog five minutes ago hasn't been included in any training data.

Solutions include: using content-based features to compute an initial embedding (a new movie's embedding can be bootstrapped from its genre, cast, and description), using an average embedding as a default, or training a model that maps entity attributes to the embedding space.

Staleness

User preferences change over time, but embeddings are typically retrained on a fixed schedule. A user who just had a baby starts searching for cribs and diapers, but their embedding still reflects their pre-baby browsing patterns. The mismatch between stale embeddings and current behavior degrades recommendation quality.

Systems handle this through more frequent retraining, real-time embedding updates via streaming pipelines, or combining static embeddings with real-time features that capture recent behavior.

Dimensionality trade-offs

Higher dimensions capture more nuance but cost more memory and compute. Lower dimensions are cheaper but may lose important distinctions. There's no universal answer, the right dimension depends on the complexity of the entity space and the downstream task.

Popularity bias is particularly insidious. Items with more interactions have more training signal, so their embeddings become better calibrated. Rare items have noisy embeddings that may land in the wrong region of the space.

This creates a feedback loop: popular items get recommended more, generating more interactions, producing better embeddings, leading to even more recommendations. Systems like YouTube address this through careful negative sampling strategies during training and explicit diversity mechanisms in serving.