{"title":"What is a Token?","description":"","content":"LLMs do not read text the way you do.\n\nWhen you type \"Hello, how are you?\" into ChatGPT, the model does not see seven words. It does not see individual letters either. It sees something in between: a sequence of tokens.\n\nTokens are the fundamental unit of how LLMs process language. They determine how much context the model can handle, how fast it generates responses, and how much you pay when using the API.\n\nYet most developers treat tokens as an afterthought. They hit context limits, get unexpectedly high bills, or wonder why the same sentence costs more in Japanese than in English.\n\nUnderstanding tokens clears up these mysteries.\n\n---\n\n# What is a Token?\n\nA token is a chunk of text that an LLM treats as a single unit.\n\nThink of tokens like syllables, but for machines. Just as we naturally break words into syllables when speaking (\"com-pu-ter\"), LLMs break text into tokens when processing.\n\nA token can be:\n\n- A whole word (\"hello\")\n- Part of a word (\"play\" + \"ing\")\n- A single character (\"!\")\n- A space or punctuation mark\n\nHere is how GPT-4 tokenizes the sentence \"Tokenization is fascinating\":\n\n\n```shell\n\"Token\" + \"ization\" + \" is\" + \" fasci\" + \"nating\"\n```\n\n\nThat is 5 tokens for 3 words. Notice how common words like \"is\" get bundled with the preceding space, while longer words get split into pieces.\n\n---\n\n# Why Tokens? Why Not Characters or Words?\n\nLLMs could process text at the character level or the word level. Each approach has trade-offs.\n\n### Character-Level Processing\n\nIf we process text one character at a time:\n\n\n```shell\n\"Hello\" → [\"H\", \"e\", \"l\", \"l\", \"o\"]\n```\n\n\n#### **Pros:**\n\n- Small vocabulary (just 256 characters for English)\n- Can handle any word, including typos and made-up words\n\n#### **Cons:**\n\n- Sequences become very long (5x longer on average)\n- Model must learn that \"H-e-l-l-o\" means \"Hello\"\n- Computationally expensive (attention scales quadratically with length)\n\n### Word-Level Processing\n\nIf we process text one word at a time:\n\n\n```shell\n\"Hello\" → [\"Hello\"]\n```\n\n\n#### **Pros:**\n\n- Sequences are short\n- Each token carries more meaning\n\n#### **Cons:**\n\n- Vocabulary explodes (English has 170,000+ words)\n- Cannot handle misspellings, new words, or rare terms\n- \"chatGPT\" and \"ChatGPT\" would be different tokens\n\n### Subword Tokenization: The Sweet Spot\n\nTokens hit the middle ground. They use subword units that are:\n\n- Large enough to be efficient (shorter sequences than characters)\n- Small enough to be flexible (smaller vocabulary than words)\n- Reusable across similar words\n\n\n```shell\n\"unhappiness\" → [\"un\", \"happiness\"]\n\"happiness\" → [\"happiness\"]\n\"happy\" → [\"happy\"]\n```\n\n\nThe prefix \"un-\" becomes its own token, reusable across \"unclear\", \"undo\", \"unfair\". The model learns its meaning once and applies it everywhere.\n\n---\n\n# How Tokenization Works\n\nTokenization happens before text ever reaches the LLM. A separate algorithm called a tokenizer converts your text into a sequence of token IDs.\n\n\n[Embed: https://link.excalidraw.com/readonly/hEF2oyWj5Fc7kNGvpBGw](https://link.excalidraw.com/readonly/hEF2oyWj5Fc7kNGvpBGw)\n\n\nThe tokenizer maintains a vocabulary: a mapping from token strings to numeric IDs. GPT-4's vocabulary contains roughly 100,000 tokens.\n\n### Step 1: Build the Vocabulary\n\nBefore the LLM is trained, the tokenizer's vocabulary is built from a large text corpus. The most common algorithm is Byte Pair Encoding (BPE).\n\n**Byte Pair Encoding (BPE)**\n\nBPE starts with individual characters and iteratively merges the most frequent pairs:\n\n1. Start with characters: [\"h\", \"e\", \"l\", \"l\", \"o\", \" \", \"w\", \"o\", \"r\", \"l\", \"d\"]\n2. Find the most common pair: (\"l\", \"l\") appears often\n3. Merge it into a new token: \"ll\"\n4. Repeat for the next most common pair\n5. Continue until you reach your target vocabulary size\n\nAfter many iterations:\n\n- \"the\" becomes a single token (very common)\n- \"hello\" might become \"hel\" + \"lo\"\n- Rare words get split into many pieces\n\n**Other Algorithms**\n\n- **WordPiece** (used by BERT): Similar to BPE but uses likelihood instead of frequency\n- **SentencePiece** (used by LLaMA): Treats the input as a raw byte stream, works well for multilingual models\n- **Unigram**: Starts with a large vocabulary and prunes it down\n\n### Step 2: Encode Text to Tokens\n\nWhen you send text to an LLM, the tokenizer encodes it:\n\n\n```python\nfrom tiktoken import encoding_for_model\n\nenc = encoding_for_model(\"gpt-4\")\ntokens = enc.encode(\"Hello, how are you?\")\nprint(tokens) # [9906, 11, 1268, 527, 499, 30]\nprint(len(tokens)) # 6 tokens\n```\n\n\nEach token ID maps to an entry in the vocabulary.\n\n### Step 3: Decode Tokens Back to Text\n\nAfter the LLM generates token IDs, the tokenizer decodes them back to text:\n\n\n```python\ntext = enc.decode([9906, 11, 1268, 527, 499, 30])\nprint(text) # \"Hello, how are you?\"\n```\n\n\nThis round-trip is lossless. Encode then decode always gives you back the original text.\n\n---\n\n# Why Tokens Matter in Practice\n\nUnderstanding tokens helps you work with LLMs more effectively.\n\n### 1. Context Windows Are Measured in Tokens\n\nWhen a model says it has a \"128K context window,\" that means 128,000 tokens, not characters or words.\n\nA rough rule of thumb for English:\n\n- 1 token is approximately 4 characters\n- 1 token is approximately 0.75 words\n- 100 tokens is approximately 75 words\n\nSo a 128K context window is roughly 96,000 words, or about 300 pages of a book.\n\nBut this varies by content:\n\n- Code uses more tokens per line (special characters, variable names)\n- Non-English text often uses more tokens per word\n- Technical writing with rare terms uses more tokens\n\n### 2. Pricing Is Per Token\n\nAPI providers charge per token, both for input (what you send) and output (what the model generates).\n\n\n```shell\nGPT-4o pricing (as of 2024):\n- Input: $2.50 per million tokens\n- Output: $10.00 per million tokens\n```\n\n\nA 1,000-word document is roughly 1,300 tokens. Processing it with GPT-4o costs about $0.003 for input.\n\nOutput tokens cost more because generation is more compute-intensive than reading.\n\n### 3. Speed Is Per Token\n\nLLMs generate text one token at a time. Generation speed is measured in tokens per second.\n\nA typical response of 500 tokens at 50 tokens/second takes 10 seconds. This is why longer responses take longer to appear.\n\nStreaming works at the token level too. Each token arrives as soon as it is generated, which is why you see text appear word-by-word.\n\n### 4. Multilingual Text Uses More Tokens\n\nEnglish text is heavily represented in LLM training data, so English words tend to be common tokens. Other languages often require more tokens to represent the same meaning.\n\nConsider the word \"hello\" in different languages:\n\n\n| ##### Language | ##### Text | ##### Tokens (GPT-4) |\n| --- | --- | --- |\n| English | hello | 1 |\n| Spanish | hola | 1 |\n| Japanese | こんにちは | 3 |\n| Arabic | مرحبا | 4 |\n| Hindi | नमस्ते | 5 |\n\n\nThis means:\n\n- Non-English users hit context limits faster\n- Non-English API calls cost more\n- Non-English generation is slower\n\nThis is an active area of research. Newer models are trained with more balanced multilingual data.\n\n### 5. Special Tokens Control Behavior\n\nLLMs use special tokens that are not part of normal text:\n\n- `<|endoftext|>`: Marks the end of a document\n- `<|im_start|>` and `<|im_end|>`: Mark the start and end of messages in chat\n- `<|tool_call|>`: Indicates the model wants to call a function\n\nThese tokens are invisible to users but critical to how the model operates. When you see a model \"stop generating,\" it has often produced an end-of-text token.\n\n---\n\n# Tokenization Quirks and Gotchas\n\nTokenization can produce surprising behavior.\n\n### Numbers Are Weird\n\nNumbers often get split in unexpected ways:\n\n\n```shell\n\"12345\" → [\"123\", \"45\"]\n\"123456\" → [\"123\", \"456\"]\n```\n\n\nThis is why LLMs struggle with arithmetic. The model does not see \"12345\" as a number. It sees two separate tokens that happen to look like digits.\n\n### Spaces Matter\n\nLeading spaces often become part of the token:\n\n\n```shell\n\"Hello\" → [\"Hello\"]\n\" Hello\" → [\" Hello\"]\n```\n\n\n\"Hello\" and \" Hello\" are different tokens. This is why formatting matters in prompts.\n\n### Case Sensitivity\n\n\n```shell\n\"Hello\" → [\"Hello\"]\n\"hello\" → [\"hello\"]\n\"HELLO\" → [\"HE\", \"LLO\"]\n```\n\n\nAll-caps text often gets split into more tokens because it is less common in training data.\n\n### Code Is Token-Heavy\n\n\n```python\ndef calculate_sum(a, b):\n return a + b\n```\n\n\nThis simple function uses many tokens:\n\n- Each punctuation mark (parentheses, colon, comma) is often its own token\n- Underscores split identifiers (\"calculate\" + \"_\" + \"sum\")\n- Indentation whitespace uses tokens\n\nCode typically uses 2-3x more tokens than prose for the same amount of information.\n\n---\n\n# How to Count Tokens\n\nYou do not need to guess. Use a tokenizer to count exactly.\n\n### For OpenAI Models (tiktoken)\n\n\n```python\nimport tiktoken\n\nenc = tiktoken.encoding_for_model(\"gpt-4\")\ntext = \"Your text here\"\ntokens = enc.encode(text)\nprint(f\"Token count: {len(tokens)}\")\n```\n\n\n### For Open Source Models (HuggingFace)\n\n\n```python\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama/Llama-2-7b\")\ntokens = tokenizer.encode(\"Your text here\")\nprint(f\"Token count: {len(tokens)}\")\n```\n\n\n### Online Tools\n\n- OpenAI Tokenizer:\n- Tiktokenizer:\n\n---\n\n# Key Takeaways\n\n1. **Tokens are subword units**: Not characters, not words, but something in between\n2. **Tokenization happens before the LLM**: A separate algorithm converts text to token IDs\n3. **Context, cost, and speed are all per-token**: Understanding tokens helps you optimize all three\n4. **Different languages use different amounts of tokens**: English is most efficient; other languages cost more\n5. **Special tokens control model behavior**: End-of-text, message boundaries, tool calls\n6. **Tokenization creates quirks**: Numbers, spaces, and code behave unexpectedly\n\n---\n\n# Further Reading\n\n- [**Tiktoken library**](undefined): OpenAI's tokenizer implementation\n- [**Hugging Face Tokenizers**](undefined): Fast tokenizers for any model\n- [**Tiktokenizer**](undefined): Visual tokenizer for experimenting\n- [**BPE Paper**](undefined): \"Neural Machine Translation of Rare Words with Subword Units\" by Sennrich et al.\n- [**SentencePiece**](undefined): Unsupervised text tokenizer\n- [**Andrej Karpathy's tokenization video**](undefined): Deep dive into BPE implementation","pageType":"ai-engineering"}

What is a Token?

Ashish Pratap Singh

Get Premium