AI systems do not compare raw words the way humans read them. They turn text, images, code, and other inputs into numbers first. An embedding is one of those numeric representations: a vector produced by a model so that useful relationships between inputs show up as distance and direction.

For AI engineers, embeddings matter because they turn messy inputs such as questions, documents, images, products, and code snippets into something software can store, compare, search, cluster, and rank.

Embeddings do something similar for model-learned similarity. An embedding model takes a piece of text, such as a word, sentence, paragraph, or document chunk, and maps it to a list of numbers. Instead of two dimensions like GPS, text embeddings usually have hundreds or thousands of dimensions.

Do not take the GPS analogy too literally. Embedding space is not a clean map where every axis has a human-readable label. It is a learned coordinate system. The useful property is relational: texts that the model judges to be similar tend to have vectors that are close under a chosen similarity metric.

The key property is that texts with similar meanings can end up near each other in this space, even when they share no important words. For example, "How do I fix a broken pipe?" and "Plumbing repair guide" may land in the same neighborhood because their meanings overlap, not because the strings look alike.

Three text inputs go into the embedding model. The two plumbing-related texts land near each other; the cake recipe lands elsewhere. This is the basis of semantic search: embed the query, embed the documents, and retrieve the documents with vectors closest to the query vector.

What Do the Dimensions Mean?

Individual dimensions do not usually have clean, human-readable meanings. They are not things like "dimension 47 = how much this text is about plumbing." The useful signal comes from the whole vector.

A single dimension may contribute to several patterns at once: topic, tone, language, intent, domain, or something harder to name. Those patterns are distributed across many dimensions, so simple labels are usually misleading.

In practice, you rarely need to interpret individual dimensions. You need to know whether the vector space behaves well for your task. For retrieval, relevant passages should rank ahead of irrelevant passages. For clustering, related items should group together. For anomaly detection, unusual items should separate from common ones. The vector is useful only if these relationships hold on your data.

You do not need to train your own embedding model to use embeddings. But understanding the rough training idea helps you anticipate where embeddings work well and where they fail.

The Training Process

Modern text embedding models are often transformer encoders or encoder-style variants. They are trained to produce nearby vectors for inputs that should match, and more distant vectors for inputs that should not.

The training data often consists of pairs or groups of related texts. These might be:

During training, the model learns from positive and negative examples. A query and a clicked or human-labeled result should move closer. A query and an unrelated document should move farther apart. This family of objectives is usually called contrastive learning, though production models often combine several objectives and data sources.

The result is a model with a useful retrieval geometry. It can place "dog" near "puppy", distinguish "bank" as a financial institution from "bank" as a river edge when context is clear, and treat "running a company" differently from "running a marathon". At query time, it applies patterns learned during training; it is not reasoning through every comparison from first principles.

From Tokens to a Single Vector

The transformer processes tokens and produces a vector for each token position. Retrieval usually needs one vector for the whole sentence, paragraph, or chunk, not one vector per token. The model has to combine the token-level vectors into a single fixed-size vector.

Common pooling strategies include mean pooling over token vectors, using a special classification token such as [CLS], or applying a model-specific pooling head. The choice matters because pooling affects what information survives in the final vector. In application code, you normally use the pooling method shipped with the model.

Generating embeddings in practice is straightforward. The important part is using the same model, dimension setting, and preprocessing everywhere you plan to compare vectors.

Using OpenAI's Embedding API

OpenAI's text-embedding-3-small produces 1536-dimensional vectors by default and is a practical baseline for many text retrieval systems. The larger text-embedding-3-large model produces 3072-dimensional vectors by default and is worth testing when retrieval quality is the bottleneck.

You send text in and receive a fixed-length list of floating-point numbers. Texts embedded with the same model and dimension setting produce vectors in the same space, which is what makes comparison meaningful.

Using Sentence Transformers Locally

If you prefer not to depend on an external API, the sentence-transformers library gives you access to many local and open-weight embedding models.

Output:

Notice the difference in dimensionality: OpenAI's small embedding model produces 1536-dimensional vectors by default, while all-MiniLM-L6-v2 produces 384-dimensional vectors. Dimension count affects storage, memory bandwidth, and search cost. It does not, by itself, prove that one model is better than another. Quality depends on the model, training data, objective, and fit to your task.

Batch Processing

In real applications, you rarely embed one text at a time. You may have thousands or millions of chunks to embed when building a search index. Both API-based models and local models support batch processing.

For API models, batch size is constrained by provider limits, token limits, and rate limits. For local models with sentence-transformers, the batch_size parameter controls how many texts are processed together on the GPU or CPU. Larger batches improve throughput until you run out of memory.

Once you have embedding vectors, you need a way to quantify how similar two of them are.

Three metrics come up often in text embedding systems: cosine similarity, dot product, and Euclidean distance. They are related, but they are not interchangeable unless you understand how your vectors are normalized.

Cosine Similarity

Cosine similarity measures the angle between two vectors, ignoring their magnitude. It answers the question: "Are these two vectors pointing in roughly the same direction?"

The formula is:

Where A . B is the dot product, and ||A|| is the magnitude, or length, of vector A. The result ranges from -1 to 1. A score near 1 means the vectors point in a similar direction. A score near 0 means they are roughly orthogonal. In real embedding systems, "near 0" does not automatically mean "unrelated"; interpretation depends on the model and data.

Think of two arrows starting from the origin. Cosine similarity cares about the angle between them. If they point in the same direction, cosine similarity is 1, even if one arrow is twice as long as the other. This makes it insensitive to vector magnitude, which is one reason cosine similarity is common for text embeddings.

The two plumbing-related sentences should have a higher similarity score than either one has with the cake recipe. The embedding model can capture the semantic relationship even though "How do I fix a broken pipe?" and "Plumbing repair guide" do not rely on exact word overlap.

Dot Product

The dot product multiplies corresponding elements and sums them. There is no normalization step.

Unlike cosine similarity, the dot product is affected by vector magnitude. A longer vector paired with another long vector can produce a larger dot product, even if the directions are not especially similar. This means the dot product mixes two signals: direction and magnitude.

That can be useful or harmful depending on your application. If magnitude encodes a meaningful signal, such as confidence or popularity, dot product can include that signal. If magnitude is just a side effect of the model, it can add noise.

Important note: If your embeddings are normalized to unit length, dot product and cosine similarity produce the same ranking. OpenAI embeddings are normalized this way. Some local models are not normalized unless you request it, so check the model documentation or normalize explicitly.

Euclidean Distance

Euclidean distance measures the straight-line distance between two points in the embedding space. It is the familiar distance formula from geometry, extended to high dimensions.

Unlike the previous two metrics where higher means more similar, with Euclidean distance, lower means more similar. Two identical vectors have a distance of zero, and the distance grows as they diverge.

The plumbing pair should have a smaller distance than the pipe-cake pair.

When to Use Which Metric

Choosing the right metric depends on the model and the index you plan to use:

The practical rule: Use the same metric during evaluation, indexing, and querying. If vectors are normalized, dot product, cosine similarity, and Euclidean distance often produce identical or near-identical rankings. If vectors are not normalized, the metric choice can change results substantially.

You cannot directly inspect a 384-dimensional or 1536-dimensional space. Dimensionality reduction methods project embeddings into 2D or 3D so you can look for clusters, outliers, and labeling errors. Treat these plots as debugging tools, not proof that the retrieval system works.

t-SNE (t-Distributed Stochastic Neighbor Embedding)

t-SNE is a common visualization algorithm. It tries to preserve local neighborhoods: points that were near each other in the original space tend to remain near each other in 2D. It does not reliably preserve global distances, so do not read too much into the space between clusters.

When you run this code, you should see topic-level grouping: programming sentences near other programming sentences, cooking near cooking, and sports near sports. The exact layout will vary because t-SNE is sensitive to initialization and parameters.

UMAP (Uniform Manifold Approximation and Projection)

UMAP is a common alternative to t-SNE. It often runs faster and tends to preserve more global structure, though it still creates a projection, not a faithful map of the original space.

t-SNE vs UMAP: When to Use Which

A practical tip: use these plots to find suspicious data, duplicated content, bad labels, and topic drift. Use retrieval metrics such as Recall@K and MRR to decide whether the embeddings are good enough for the product.

Before you start building with embeddings, a few practical details are worth getting right early.

Dimensionality

Embedding models produce vectors of different sizes. Common dimensions include 384 (MiniLM), 768 (BERT-base), 1024 (many modern models), and 1536 (OpenAI's text-embedding-3-small). OpenAI's text-embedding-3-large goes up to 3072 dimensions.

Higher dimensionality can give a model more capacity, but dimension count is not a quality guarantee. It does create predictable costs:

Some models are trained so shorter versions of the vector remain useful. This is often described as Matryoshka-style dimensionality. OpenAI's text-embedding-3 models expose this through the dimensions parameter, letting you trade some quality for lower storage and latency without switching models.

Normalization

Normalization means scaling a vector so its magnitude, or length, equals 1. This matters because cosine similarity and dot product give the same ranking for normalized vectors, and many vector indexes are configured with that assumption.

Do not assume every provider or local model normalizes outputs. If your retrieval setup expects cosine-like behavior, normalize explicitly unless the model documentation says the vectors are already unit length:

Batch Processing and Rate Limits

When you have thousands or millions of texts to embed, you need a strategy for efficient batch processing.

For API-based embedding, the main constraints are:

For local models with sentence-transformers, GPU or CPU memory is usually the bottleneck. Larger batch sizes improve throughput until you run out of memory.

For API-based embedding, use bounded retries and keep failed batches visible. The example below is intentionally simple; production jobs should also log failures, track progress, and be safe to resume.

The diagram shows the usual production pipeline. Raw text is cleaned, split into batches, embedded, optionally normalized, and stored with metadata in a vector database. The metadata is not an afterthought; it is how you enforce access control, filter by tenant or source, and trace a retrieved chunk back to the original document.

Caching Embeddings

Generating embeddings costs money, compute, or both. Do not re-embed text that has not changed. Store the embedding with the source text hash, embedding model name, dimension setting, normalization choice, and chunking version. If any of those change, treat the cached vector as stale.

References