{"title":"Tokenization","description":"","content":"Before a language model can understand or generate text, it must first convert that text into a form it can process. Neural networks do not work directly with characters or words. They operate on numbers. Tokenization is the step that bridges this gap.\n\nTokenization is the process of breaking raw text into smaller units called **tokens** and mapping those tokens to numeric IDs that the model can understand. A token might be a word, part of a word, a punctuation mark, or even a single character, depending on the tokenizer.\n\nFor example, the sentence: \"LLMs are transforming software development\" might be broken into tokens like:\n\n`[\"LL\", \"Ms\", \" are\", \" transforming\", \" software\", \" development\", \".\"]`\n\nEach token is then converted into a number and fed into the model.\n\nIn this chapter, we will explore how tokenization works, why modern models use subword tokens instead of full words, and how tokenization affects context length, cost, and model performance.\n\n---\n\n# Why Not Just Use Words?\n\nThe first question worth asking is: why do LLMs need tokens at all? Why not just work with words directly?\n\nThere are three fundamental problems with using words as the basic unit.\n\n#### **The vocabulary problem**\n\nEnglish has roughly 170,000 words in current use. Add technical jargon, proper nouns, misspellings, slang, and other languages, and you are looking at millions of unique strings. Every word needs its own entry in the model's vocabulary, and every entry comes with parameters the model has to learn. A vocabulary of millions would make training impossibly expensive.\n\n#### **The unknown word problem**\n\nNo matter how large your vocabulary is, users will always type something the model has never seen. A new product name, a typo, a word from a language that was underrepresented in training data. A word-level model has no way to handle these. It can only shrug and output an \"unknown\" placeholder.\n\n#### **The morphology problem**\n\nWords like \"run\", \"running\", \"runner\", and \"runs\" are clearly related, but a word-level model treats them as completely independent entries. It has to learn the meaning of each one separately, wasting capacity on patterns that are obvious to humans.\n\nTokens solve all three problems. Instead of splitting text at word boundaries, tokenization algorithms find subword units, chunks that are smaller than words but larger than individual characters. Common words like \"the\" stay as single tokens. Rare words get split into pieces. The word \"unhappiness\" might become [\"un\", \"happiness\"], or [\"un\", \"happi\", \"ness\"], depending on the algorithm.\n\nThis gives you the best of both worlds: a manageable vocabulary size (typically 32,000 to 100,000 entries) that can still represent any possible input, including words it has never seen before.\n\n\n```mermaid\nflowchart TD\n INPUT[\"Input: 'unhappiness'\"]:::primary\n\n subgraph Strategies\n WORD[\"Word-Level
['unhappiness']
1 token\"]:::orange\n CHAR[\"Character-Level
['u','n','h','a','p',...]
11 tokens\"]:::red\n SUB[\"Subword (BPE)
['un','happiness']
2 tokens\"]:::green\n end\n\n INPUT --> WORD\n INPUT --> CHAR\n INPUT --> SUB\n\n WORD --> W_ISSUE[\"Huge vocabulary
Can't handle unknown words\"]:::red\n CHAR --> C_ISSUE[\"Too many tokens
Slow, loses word meaning\"]:::red\n SUB --> S_WIN[\"Small vocabulary
Handles any input\"]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nCharacter-level tokenization can handle any input, but it produces very long sequences (each character is a token), which makes the model slow and makes it harder to learn word-level meaning. Word-level tokenization is compact but brittle. Subword tokenization hits the sweet spot.\n\n---\n\n# Byte Pair Encoding (BPE)\n\nByte Pair Encoding is the most widely used tokenization algorithm in modern LLMs. GPT-2, GPT-3, GPT-4, LLaMA, and Mistral all use variants of BPE. The algorithm is surprisingly simple, and understanding it gives you strong intuition for how tokenization behaves in practice.\n\n### The Core Idea\n\nBPE starts with the smallest possible vocabulary: individual bytes (or characters). Then it repeatedly finds the most frequent pair of adjacent tokens in the training data and merges them into a single new token. It keeps doing this until the vocabulary reaches a target size.\n\nThat is the entire algorithm. Let's walk through it step by step.\n\n### BPE Step by Step\n\nImagine we have a tiny training corpus with just these words (with frequencies):\n\n\n| Word | Frequency |\n|------|-----------|\n| low | 5 |\n| lower | 2 |\n| newest | 6 |\n| widest | 3 |\n\n\n#### **Step 1: Start with characters**\n\nWe split every word into individual characters and add a special end-of-word marker (let's use `_`). Our initial vocabulary is every unique character in the corpus.\n\n\n```plaintext\nInitial tokens:\nl o w _ (frequency: 5)\nl o w e r _ (frequency: 2)\nn e w e s t _ (frequency: 6)\nw i d e s t _ (frequency: 3)\n\nInitial vocabulary: {l, o, w, e, r, n, s, t, i, d, _}\n```\n\n\n#### **Step 2: Count all adjacent pairs**\n\nWe look at every pair of neighboring tokens across all words, weighted by frequency:\n\n\n```plaintext\nPair counts:\n(e, s) = 6 + 3 = 9 (from \"newest\" and \"widest\")\n(s, t) = 6 + 3 = 9 (from \"newest\" and \"widest\")\n(t, _) = 6 + 3 = 9 (from \"newest\" and \"widest\")\n(l, o) = 5 + 2 = 7 (from \"low\" and \"lower\")\n(o, w) = 5 + 2 = 7 (from \"low\" and \"lower\")\n(n, e) = 6 (from \"newest\")\n(e, w) = 6 (from \"newest\")\n(w, _) = 5 (from \"low\")\n...\n```\n\n\n#### **Step 3: Merge the most frequent pair**\n\nThe pairs (e, s), (s, t), and (t, _) are all tied at 9. We pick one, say (e, s), and merge it into a new token `es`.\n\n\n```plaintext\nAfter merging (e, s) -> es:\nl o w _ (frequency: 5)\nl o w e r _ (frequency: 2)\nn e w es t _ (frequency: 6)\nw i d es t _ (frequency: 3)\n\nVocabulary: {l, o, w, e, r, n, s, t, i, d, _, es}\n```\n\n\n#### **Step 4: Repeat**\n\nNow we count pairs again with the updated tokens. The pair (es, t) appears 9 times. Merge it into `est`.\n\n\n```plaintext\nAfter merging (es, t) -> est:\nl o w _ (frequency: 5)\nl o w e r _ (frequency: 2)\nn e w est _ (frequency: 6)\nw i d est _ (frequency: 3)\n\nVocabulary: {l, o, w, e, r, n, s, t, i, d, _, es, est}\n```\n\n\nNext, (est, ) appears 9 times. Merge into `est`.\n\nThen (l, o) appears 7 times. Merge into `lo`.\n\nThen (lo, w) appears 7 times. Merge into `low`.\n\nAnd so on, until we hit our target vocabulary size.\n\n\n```mermaid\nflowchart LR\n A[\"Characters
l,o,w,e,s,t...\"]:::primary\n B[\"Merge (e,s)
→ es\"]:::orange\n C[\"Merge (es,t)
→ est\"]:::orange\n D[\"Merge (est,_)
→ est_\"]:::orange\n E[\"Merge (l,o)
→ lo\"]:::teal\n F[\"Merge (lo,w)
→ low\"]:::teal\n G[\"Final
Vocabulary\"]:::green\n\n A --> B --> C --> D --> E --> F --> G\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n---\n\n### BPE Simulation\n\n\n\n\n\n---\n\n### Why BPE Works Well\n\nThe beauty of BPE is that it is purely data-driven. It does not need any linguistic knowledge. Common words like \"the\" get merged into single tokens early because their character pairs are extremely frequent. Rare or technical words get split into smaller pieces that the model can still process.\n\nThis means that:\n\n- **Common words** are single tokens (efficient, cheap)\n- **Rare words** are split into known subwords (still representable)\n- **Completely new words** fall back to character-level tokens (never truly \"unknown\")\n\nThe vocabulary size is a hyperparameter that model creators choose. GPT-2 uses about 50,257 tokens. GPT-4 uses roughly 100,000. LLaMA uses 32,000. Larger vocabularies mean fewer tokens per text (more efficient), but more parameters to train.\n\n---\n\n# WordPiece\n\nWordPiece is another subword tokenization algorithm that works similarly to BPE but with a key difference in how it chooses which pair to merge. BERT, DistilBERT, and other models from Google's ecosystem use WordPiece.\n\n### How It Differs from BPE\n\nBPE merges the pair that appears most frequently in the corpus. WordPiece merges the pair that maximizes the likelihood of the training data. In simpler terms, WordPiece asks: \"Which merge would make the training corpus most probable under our current model?\"\n\nThe practical difference is subtle. BPE favors pairs that are common in absolute terms. WordPiece favors pairs where the combination appears much more often than you would expect given the individual frequencies of each piece. If token A appears 1,000 times and token B appears 1,000 times, but AB appears 900 times, WordPiece considers that a very strong signal to merge, even if another pair appears 950 times in absolute terms.\n\n### The ## Prefix Convention\n\nWordPiece uses a special prefix `##` to mark tokens that are continuations of a previous token (i.e., not the start of a word). For example:\n\n\n```plaintext\nWord: \"tokenization\"\n\nBPE output: [\"token\", \"ization\"]\nWordPiece output: [\"token\", \"##ization\"]\n```\n\n\nThe `##` prefix tells the model \"this piece is attached to the previous token, not a standalone word.\" This helps the model distinguish between the word \"a\" (the article) and the \"a\" at the end of \"extra\".\n\n### BPE vs WordPiece\n\n\n| Aspect | BPE | WordPiece |\n|--------|-----|-----------|\n| Merge criterion | Most frequent pair | Highest likelihood gain |\n| Prefix marking | No prefix convention | Uses `##` for continuations |\n| Used by | GPT-2/3/4, LLaMA, Mistral | BERT, DistilBERT, Electra |\n| Vocabulary size | 32K - 100K typical | 30K typical |\n| Speed | Very fast | Slightly slower (likelihood calc) |\n\n\nFor most practical work as an AI engineer, the difference between BPE and WordPiece rarely matters. What matters is knowing that different models use different tokenizers, so the same text produces different token counts and different token boundaries. This directly affects cost and behavior.\n\n---\n\n# Tokenization in Practice: Using tiktoken\n\nEnough theory. Let's get our hands dirty with actual tokenization libraries.\n\nOpenAI's `tiktoken` library is the go-to tool for working with GPT tokenizers. It is fast (written in Rust under the hood) and straightforward to use.\n\n### Installation and Basic Usage\n\n\n**main.py**\n\n```python\nimport tiktoken\n\n# Get the tokenizer for GPT-4 / GPT-4o\nencoder = tiktoken.encoding_for_model(\"gpt-4o\")\n\ntext = \"Tokenization is surprisingly important for AI engineers.\"\n\n# Encode text to token IDs\ntoken_ids = encoder.encode(text)\nprint(f\"Token IDs: {token_ids}\")\nprint(f\"Number of tokens: {len(token_ids)}\")\n\n# Decode back to text\ndecoded = encoder.decode(token_ids)\nprint(f\"Decoded: {decoded}\")\n\n# See individual tokens as strings\ntokens = [encoder.decode([tid]) for tid in token_ids]\nprint(f\"Tokens: {tokens}\")\n```\n\n\nNotice a few things after running this. \"Tokenization\" gets split into \"Token\" and \"ization\", which makes sense since \"Token\" is a common subword. Spaces are attached to the beginning of words, not treated as separate tokens. The period is its own token. \"AI\" stays as a single token because it is extremely common in the training data.\n\n### Comparing Tokenizers Across Models\n\nDifferent models use different tokenizers with different vocabularies. This means the same text produces different token counts. Let's see this in action.\n\n\n**main.py**\n\n```python\nimport tiktoken\n\ntext = \"The quick brown fox jumps over the lazy dog.\"\n\n# GPT-4o uses cl100k_base encoding\ngpt4_enc = tiktoken.encoding_for_model(\"gpt-4o\")\n\n# GPT-3.5 also uses cl100k_base\ngpt35_enc = tiktoken.encoding_for_model(\"gpt-3.5-turbo\")\n\n# Older GPT-2 uses a different encoding\ngpt2_enc = tiktoken.get_encoding(\"gpt2\")\n\nprint(f\"GPT-4o tokens: {len(gpt4_enc.encode(text))} tokens\")\nprint(f\"GPT-3.5 tokens: {len(gpt35_enc.encode(text))} tokens\")\nprint(f\"GPT-2 tokens: {len(gpt2_enc.encode(text))} tokens\")\n```\n\n\nFor simple English text, the differences are usually small. But try it with code, JSON, or non-English text, and the gaps become significant. Let's see that.\n\n\n**main.py**\n\n```python\nimport tiktoken\n\ngpt4_enc = tiktoken.encoding_for_model(\"gpt-4o\")\ngpt2_enc = tiktoken.get_encoding(\"gpt2\")\n\ntest_cases = {\n \"English\": \"The quick brown fox jumps over the lazy dog.\",\n \"Python code\": \"def fibonacci(n):\\n if n <= 1:\\n return n\\n return fibonacci(n-1) + fibonacci(n-2)\",\n \"Japanese\": \"Large Language Modelは自然言語処理の分野で革命を起こしました。\",\n \"JSON\": '{\"name\": \"Alice\", \"age\": 30, \"scores\": [95, 87, 92]}',\n \"Emoji\": \"I love coding! 🚀💻🎉 Let's build something amazing! 🔥\",\n}\n\nprint(f\"{'Input':<15} {'GPT-4o':<10} {'GPT-2':<10} {'Ratio':<10}\")\nprint(\"-\" * 45)\nfor label, text in test_cases.items():\n gpt4_count = len(gpt4_enc.encode(text))\n gpt2_count = len(gpt2_enc.encode(text))\n ratio = gpt2_count / gpt4_count\n print(f\"{label:<15} {gpt4_count:<10} {gpt2_count:<10} {ratio:<10.2f}\")\n```\n\n\nYou will see that GPT-2's tokenizer produces significantly more tokens for non-English text and special characters. GPT-4o's tokenizer has been trained on a more diverse dataset, so it handles these cases more efficiently. Fewer tokens means lower cost and more room within the context window.\n\n### Using Hugging Face Tokenizers\n\nFor non-OpenAI models, the `transformers` library from Hugging Face gives you access to their tokenizers.\n\n\n**main.py**\n\n```python\nfrom transformers import AutoTokenizer\n\n# Load the LLaMA tokenizer\nllama_tokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Llama-2-7b-hf\")\n\n# Load BERT's WordPiece tokenizer for comparison\nbert_tokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n\ntext = \"Tokenization algorithms are fascinating.\"\n\n# Tokenize with LLaMA (BPE-based)\nllama_tokens = llama_tokenizer.tokenize(text)\nprint(f\"LLaMA tokens: {llama_tokens}\")\nprint(f\"LLaMA count: {len(llama_tokens)}\")\n\n# Tokenize with BERT (WordPiece)\nbert_tokens = bert_tokenizer.tokenize(text)\nprint(f\"BERT tokens: {bert_tokens}\")\nprint(f\"BERT count: {len(bert_tokens)}\")\n```\n\n\n#### **Output:**\n\n\n```plaintext\nLLaMA tokens: ['▁Token', 'ization', '▁algorithms', '▁are', '▁fasci', 'nating', '.']\nLLaMA count: 7\nBERT tokens: ['token', '##ization', 'algorithms', 'are', 'fascinating', '.']\nBERT count: 6\n```\n\n\nNotice how LLaMA uses `▁` (a special Unicode character) to mark the beginning of words, while BERT uses `##` to mark continuations. Both are solving the same problem, \"where do word boundaries fall?\", but with different conventions.\n\n---\n\n# The Multilingual Problem\n\nTokenization gets interesting (and frustrating) when you move beyond English. This is not an edge case. If your application serves a global audience, tokenization quirks directly affect cost and user experience.\n\n### Why Non-English Text Uses More Tokens\n\nBPE tokenizers are trained on large corpora. These corpora are heavily weighted toward English. As a result, the tokenizer learns efficient representations for English words but not for words in other languages.\n\nA single English word like \"hello\" is typically one token. But the equivalent word in Hindi, \"नमस्ते\" (namaste), might be three or four tokens. The Chinese word for \"hello\", \"你好\", might be two tokens with a modern tokenizer but could be six or more with an older one.\n\nThis has real consequences:\n\n- **Cost:** A Japanese user's request costs 2-3x more tokens than the equivalent English request, even though the semantic content is the same.\n- **Context limits:** That 128K token context window holds far less Japanese text than English text. A document that fits comfortably in English might get truncated in Korean.\n- **Response quality:** When the model uses more tokens to represent the input, it has fewer tokens left for generating the output, which can lead to shorter or lower-quality responses.\n\n\n```mermaid\nflowchart TB\n subgraph English\n\t\tdirection TB\n E_IN[\"'Hello, how are you?'\"]:::green\n E_TOK[\"5 tokens\"]:::green\n end\n\n subgraph Japanese\n\t\tdirection TB\n J_IN[\"'こんにちは、お元気ですか？'\"]:::red\n J_TOK[\"13 tokens\"]:::red\n end\n\n subgraph Hindi\n\t\tdirection TB\n H_IN[\"'नमस्ते, आप कैसे हैं?'\"]:::red\n H_TOK[\"17 tokens\"]:::red\n end\n\n E_IN --> E_TOK\n J_IN --> J_TOK\n H_IN --> H_TOK\n\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nThe same greeting in three languages produces vastly different token counts. The exact numbers depend on the tokenizer, but the pattern is consistent: English is almost always the most efficient.\n\n### Newer Models Are Getting Better\n\nIt is worth noting that this gap has been shrinking. GPT-4o's tokenizer was specifically trained on more multilingual data, and it handles non-Latin scripts significantly better than GPT-2 or even GPT-3.5. LLaMA 3's tokenizer also expanded its vocabulary to better cover non-English languages.\n\nBut the gap has not closed entirely. If you are building a multilingual application, you should always test token counts across your target languages and factor the overhead into your cost estimates.\n\n---\n\n# Token Counting and Cost Estimation\n\nUnderstanding tokenization is not just academic. It directly affects how much money you spend. Every LLM API charges by the token, and input tokens and output tokens often have different prices.\n\n### The Cost Formula\n\nLLM providers publish prices in \"dollars per million tokens.\" To estimate the cost of a single request:\n\n\n```plaintext\nCost = (input_tokens x input_price_per_token) + (output_tokens x output_price_per_token)\n```\n\n\nLet's make this concrete with a Python function.\n\n\n**main.py**\n\n```python\nimport tiktoken\n\ndef estimate_cost(\n prompt: str,\n expected_output_tokens: int = 500,\n model: str = \"gpt-4o\",\n input_price_per_million: float = 2.50,\n output_price_per_million: float = 10.00,\n) -> dict:\n \"\"\"Estimate the cost of an LLM API call.\"\"\"\n encoder = tiktoken.encoding_for_model(model)\n input_tokens = len(encoder.encode(prompt))\n\n input_cost = (input_tokens / 1_000_000) * input_price_per_million\n output_cost = (expected_output_tokens / 1_000_000) * output_price_per_million\n total_cost = input_cost + output_cost\n\n return {\n \"input_tokens\": input_tokens,\n \"expected_output_tokens\": expected_output_tokens,\n \"input_cost\": f\"${input_cost:.6f}\",\n \"output_cost\": f\"${output_cost:.6f}\",\n \"total_cost\": f\"${total_cost:.6f}\",\n }\n\n# Example: Estimating cost for a customer support prompt\nprompt = \"\"\"You are a helpful customer support agent for an e-commerce company.\nThe customer has an order that was shipped 5 days ago but hasn't arrived.\nTheir order number is #12345. Please help them check the status and offer\nappropriate solutions.\"\"\"\n\nresult = estimate_cost(prompt, expected_output_tokens=300)\nfor key, value in result.items():\n print(f\"{key}: {value}\")\n```\n\n\nA single request is cheap. The cost adds up when you scale. If your application handles 100,000 requests per day, even small inefficiencies in prompt design matter. Shaving 100 tokens off your system prompt across 100K daily requests saves you real money over a month.\n\n### Context Window Math\n\nEvery model has a context window, the maximum number of tokens it can process in a single request (input + output combined). Here are some common limits:\n\n\n| Model | Context Window |\n|-------|---------------|\n| GPT-4o | 128,000 tokens |\n| Claude 3.5 Sonnet | 200,000 tokens |\n| LLaMA 3.1 (8B) | 128,000 tokens |\n| Gemini 1.5 Pro | 1,000,000 tokens |\n| GPT-3.5 Turbo | 16,384 tokens |\n\n\nThe context window must fit your entire input (system prompt + conversation history + user message) plus the model's output. If you set `max_tokens` to 4,000 and your input is 125,000 tokens, you need a context window of at least 129,000 tokens.\n\nThis is why tokenization matters for architecture decisions. If you are building a RAG system that stuffs retrieved documents into the prompt, you need to know exactly how many tokens those documents consume. Estimating by word count is unreliable because the words-to-tokens ratio varies by content type.\n\n\n**main.py**\n\n```python\nimport tiktoken\n\ndef check_context_fit(\n system_prompt: str,\n user_message: str,\n retrieved_docs: list[str],\n max_output_tokens: int = 2000,\n context_window: int = 128000,\n model: str = \"gpt-4o\",\n) -> dict:\n \"\"\"Check if everything fits within the model's context window.\"\"\"\n encoder = tiktoken.encoding_for_model(model)\n\n system_tokens = len(encoder.encode(system_prompt))\n user_tokens = len(encoder.encode(user_message))\n doc_tokens = sum(len(encoder.encode(doc)) for doc in retrieved_docs)\n\n total_input = system_tokens + user_tokens + doc_tokens\n total_needed = total_input + max_output_tokens\n fits = total_needed <= context_window\n\n return {\n \"system_prompt_tokens\": system_tokens,\n \"user_message_tokens\": user_tokens,\n \"document_tokens\": doc_tokens,\n \"total_input_tokens\": total_input,\n \"max_output_tokens\": max_output_tokens,\n \"total_needed\": total_needed,\n \"context_window\": context_window,\n \"fits\": fits,\n \"remaining_tokens\": context_window - total_needed if fits else 0,\n }\n```\n\n\nThis kind of utility function is something you will use constantly in production AI applications. It is much better to check before sending a request than to get a truncation error back from the API.\n\n---\n\n### References\n\n- [OpenAI tiktoken library (GitHub)](https://github.com/openai/tiktoken)\n- [Hugging Face Tokenizers documentation](https://huggingface.co/docs/tokenizers)\n- [Sennrich, Haddow, Birch - \"Neural Machine Translation of Rare Words with Subword Units\" (BPE paper)](https://arxiv.org/abs/1508.07909)\n- [OpenAI Tokenizer (interactive tool)](https://platform.openai.com/tokenizer)\n- [Google's WordPiece paper - \"Japanese and Korean Voice Search\"](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37842.pdf)","pageType":"ai-engineering"}

Tokenization

Ashish Pratap Singh

Get Premium