Before a language model can process text, the text has to be converted into numbers. Tokenization is the step that does that conversion.

Tokenization splits raw text into smaller units called tokens and maps each token to a numeric ID from the model's vocabulary. Depending on the tokenizer, a token might be a word, part of a word, punctuation, whitespace, a code symbol, an emoji, a byte sequence, or a single character.

For example, the sentence:

might be split into tokens like:

["LL", "Ms", " are", " transforming", " software", " development", "."]

Each token is converted into an integer ID. The model then maps those IDs to embeddings and processes the sequence.

This chapter explains how tokenization works, why modern models use subword and byte-aware tokenizers, and why tokenization affects context length, cost, latency, retrieval, multilingual behavior, and production reliability.

Why do LLMs need tokens at all? Why not use words directly?

There are three practical problems with using words as the basic unit.

The vocabulary problem

Real user input does not come from a clean dictionary. It includes technical terms, product names, URLs, filenames, code, punctuation, slang, misspellings, identifiers, and many languages. A word-level vocabulary would either become extremely large or fail on ordinary inputs.

The unknown word problem

No matter how large the vocabulary is, users will type strings the tokenizer has never seen: a new library name, a random order ID, a typo, a domain-specific acronym, or a word from an underrepresented language. A strict word-level tokenizer has to map those strings to an unknown token or lose information.

The morphology problem

Words such as "run", "running", "runner", and "runs" are related. So are "serialize", "serializer", and "deserialization". A word-level tokenizer treats each surface form as a separate vocabulary item. The model can still learn relationships from context, but the tokenizer is not helping it reuse the shared pieces.

Subword tokenization is the practical compromise. Instead of splitting only at word boundaries, the tokenizer learns reusable pieces that are smaller than many words but larger than individual characters. Common words like "the" may stay as single tokens. Rare words get split into pieces. The word "unhappiness" might become ["un", "happiness"] or ["un", "happi", "ness"], depending on the tokenizer.

This gives the model a manageable vocabulary, often tens or hundreds of thousands of tokens, while still handling rare and new strings. Byte-level and byte-fallback tokenizers can represent arbitrary text without a dedicated unknown-word token. Some older tokenizers can still emit an unknown token, which is one reason tokenizer choice matters.

Character-level tokenization can handle arbitrary input, but it creates long sequences and makes the model spend many steps reconstructing word-level patterns. Word-level tokenization is compact for common words but brittle on real input. Subword tokenization usually gives the best engineering tradeoff.

Byte Pair Encoding (BPE) is one of the main tokenization families used by modern language models. GPT-style tokenizers, Llama tokenizers, and several open model families use BPE or BPE-like variants, often with byte-level handling or SentencePiece conventions. The exact implementation differs by model, but the core idea is straightforward.

The Core Idea

BPE starts with a small vocabulary, often individual bytes or characters. It repeatedly finds a frequent pair of adjacent tokens in the training corpus and merges that pair into a new token. It continues until the vocabulary reaches the target size.

That merge rule creates a vocabulary where common strings become short token sequences and rare strings fall back to smaller pieces.

BPE Step by Step

Imagine a tiny training corpus with these words and frequencies:

Step 1: Start with characters

We split every word into individual characters and add a special end-of-word marker. In this toy example, we use _. The initial vocabulary is every unique character in the corpus.

Step 2: Count all adjacent pairs

We count neighboring token pairs across all words, weighted by word frequency:

Step 3: Merge the most frequent pair

The pairs (e, s), (s, t), and (t, _) are tied at 9. We pick one, say (e, s), and merge it into a new token, es.

Step 4: Repeat

Now we count pairs again with the updated tokens. The pair (es, t) appears 9 times. Merge it into est.

Next, (est, _) appears 9 times. Merge it into est_.

Then (l, o) appears 7 times. Merge it into lo.

Then (lo, w) appears 7 times. Merge it into low.

The process continues until the tokenizer reaches its target vocabulary size.

BPE Simulation

Why BPE Works Well

BPE is data-driven. It does not need a hand-written grammar or language-specific dictionary. Common strings like "the", "ing", ": ", or "def " can become efficient tokens because they appear often. Rare or technical strings get split into smaller known pieces.

In practice:

Vocabulary size is a model-design choice. GPT-2 used about 50K tokens. Many newer OpenAI models use cl100k_base or o200k_base tokenizers with roughly 100K to 200K tokens. Llama 2 used a 32K-token SentencePiece vocabulary; Llama 3 expanded to a much larger vocabulary to improve multilingual and code efficiency. Larger vocabularies can reduce token counts, but they also increase embedding and output-layer size and can change model behavior around rare strings.

WordPiece is another subword tokenization algorithm. BERT, DistilBERT, and many older encoder-style models use WordPiece.

How It Differs from BPE

BPE is usually explained as merging frequent adjacent pairs. WordPiece chooses tokens using a likelihood-based objective: it prefers pieces that improve the probability of the training corpus under the tokenizer's model.

The practical difference is subtle for most application engineers. BPE tends to favor pairs that are common in absolute terms. WordPiece favors pieces that are informative relative to their components. If token A and token B are common separately but appear together far more often than expected, WordPiece treats that as a useful signal.

The ## Prefix Convention

WordPiece commonly uses the prefix ## to mark tokens that continue a word rather than start one. For example:

The ## prefix marks a piece as attached to the previous token. That helps distinguish a standalone token from the same characters inside a larger word.

BPE vs WordPiece

For most AI engineering work, the algorithm name matters less than the operational fact: different models use different tokenizers. The same text can produce different token counts, different token boundaries, and different failure modes.

A good way to build intuition is to inspect real tokenization output.

OpenAI's tiktoken library is a common tool for working with OpenAI tokenizers. It is fast and exposes tokenizer families such as cl100k_base and o200k_base.

Installation and Basic Usage

When you run this, look at the token strings, not only the count. You will see that spaces are often attached to the beginning of words, punctuation may be separate, and common strings such as "AI" may be compact tokens. Do not assume a word always maps to one token. Inspect the tokenizer for the model you plan to use.

Comparing Tokenizers Across Models

Different models use different tokenizers with different vocabularies. This means the same text can produce different token counts. Here is a small comparison.

For simple English text, differences are often small. With code, JSON, emoji, mixed scripts, or non-English text, the gaps can be much larger.

Older tokenizers often produce more tokens for non-English text, emoji, and some structured data, while newer tokenizers tend to have broader multilingual and code coverage. Fewer tokens usually means lower cost, lower latency, and more room in the context window.

Using Hugging Face Tokenizers

For many non-OpenAI models, the transformers library from Hugging Face gives you access to the tokenizer packaged with the model.

Example output:

Token boundary markers vary. Some SentencePiece tokenizers use ▁ to mark word starts. GPT-style tokenizers often expose space-aware tokens such as Ġ or tokens that decode with a leading space. BERT uses ## to mark continuations. These conventions solve the same practical problem: preserving word-boundary information after text has been split into subword pieces.

Tokenization becomes especially important when you move beyond English. For products with a global audience, tokenizer behavior affects cost, latency, truncation, and sometimes quality.

Why Non-English Text Uses More Tokens

Tokenizers are trained on text corpora. If the tokenizer training data is heavily weighted toward English, English strings get efficient representations and other scripts may be split into more pieces.

A single English word like "hello" is often one token. The Hindi greeting "नमस्ते" may be multiple tokens. The Chinese greeting "你好" may be compact with a modern multilingual tokenizer and much less compact with an older tokenizer.

This has real consequences:

The exact numbers depend on the tokenizer. Do not rely on a universal multiplier. Measure token counts for the languages and content types your application serves.

Newer Models Are Getting Better

This gap has narrowed. Newer tokenizers such as OpenAI's o200k_base and Llama 3's expanded vocabulary handle many non-English and code-heavy inputs more efficiently than older GPT-2-era tokenizers.

The gap has not disappeared. If you are building a multilingual application, test token counts across your target languages and include that overhead in cost estimates, prompt budgets, and retrieval chunk sizes.

Tokenization directly affects cost. Most LLM APIs charge by token, and input, cached input, output, reasoning, image, and audio tokens may be priced differently depending on the provider and model.

The Cost Formula

Providers commonly publish text prices in dollars per million tokens. For a text-only request, the rough cost is:

In code, keep prices configurable. Model pricing changes, and hard-coding old rates into application logic is a good way to produce wrong forecasts.

A single request may be cheap. Cost becomes visible at scale. If your application handles 100,000 requests per day, a 100-token system-prompt reduction removes 10 million input tokens per day. Whether that is worth doing depends on model price, cache hit rate, latency goals, and how much clarity you lose by shortening the prompt.

Context Window Math

Every model has a context window: the maximum number of tokens it can process in a single request. The budget must cover the input plus the tokens the model will generate. Some APIs also count hidden reasoning tokens, tool-call arguments, image tokens, or wrapper formatting, depending on the model.

Context sizes change often, so treat the following as examples, not as a permanent reference:

The context window must fit the entire request: system instructions, developer instructions, conversation history, retrieved documents, tool results, the user message, and the model's output. If you reserve 4,000 output tokens and your input is 125,000 tokens, you need at least 129,000 tokens of usable context.

This is why tokenization affects architecture. A RAG system needs chunk sizes, retrieval limits, prompt templates, citation metadata, and output budgets that fit the target model. Estimating by word count is unreliable because the word-to-token ratio varies by language and content type.

This kind of utility belongs near your prompt-building code. It is better to reject, summarize, or retrieve less context before the request than to discover the problem through truncation or an API error.

Tokenization is easy to ignore until it breaks something. Watch for these issues:

A good production habit is to count tokens with the same tokenizer as the model you will call, measure representative inputs, and leave margin for output and provider-side formatting.

References