{"title":"How Are LLMs Trained?","description":"","content":"# How Are LLMs Trained?\n\n**Title:** How Are LLMs Trained? **Type:** Concept Explainer **Target Length:** 4,000-5,000 words **Audience:** Software engineers learning AI/ML\n\n---\n\n## Title Options\n\n1. How Are LLMs Trained?\n2. The Complete Guide to LLM Training\n3. From Raw Text to ChatGPT: How LLMs Learn\n4. Inside LLM Training: Pre-training, Fine-tuning, and RLHF\n\n**Selected Title:** How Are LLMs Trained?\n\n**Subject Line:** The three-stage process that turns internet text into an AI assistant.\n\n---\n\nTraining a large language model is one of the most expensive and complex engineering efforts in modern computing.\n\nGPT-4 reportedly cost over $100 million to train. It required thousands of GPUs running for months, processing trillions of words from the internet. The result is a model with hundreds of billions of parameters that can write code, explain physics, and hold conversations.\n\nBut how does this actually work?\n\nIn this article, we will break down the complete LLM training pipeline: from raw internet text to a helpful AI assistant. By the end, you will understand each stage of training and why it matters.\n\n---\n\n## The Three Stages of LLM Training\n\nTraining an LLM like GPT-4 or Claude happens in three distinct stages:\n\n\n```shell\n+------------------+ +------------------+ +------------------+\n| STAGE 1 | | STAGE 2 | | STAGE 3 |\n| Pre-training | --> | Supervised | --> | RLHF |\n| | | Fine-tuning | | |\n+------------------+ +------------------+ +------------------+\n | | |\n v v v\n Base Model SFT Model Assistant\n (Internet (Follows (Helpful,\n simulator) instructions) aligned)\n```\n\n\n**Stage 1: Pre-training** - The model learns language by predicting the next token on trillions of words from the internet. This creates a \"base model\" that can complete text but is not helpful or safe.\n\n**Stage 2: Supervised Fine-Tuning (SFT)** - The model learns to follow instructions and behave like an assistant by training on human-written examples of ideal responses.\n\n**Stage 3: RLHF** - The model is further refined using human preferences, learning to generate responses that humans rate as helpful, harmless, and honest.\n\nLet us examine each stage in detail.\n\n---\n\n## Stage 1: Pre-training\n\nPre-training is the most expensive and foundational stage. This is where the model learns language itself.\n\n### Step 1: Collect Training Data\n\nLLMs are trained on massive datasets scraped from the internet. Sources include:\n\n- Web pages (via Common Crawl)\n- Books and academic papers\n- Wikipedia\n- Code repositories (GitHub)\n- Forums and social media\n- News articles\n\nThe scale is staggering:\n\n\n| ##### Model | ##### Training Tokens |\n| --- | --- |\n| GPT-3 | 300 billion |\n| LLaMA 2 | 2 trillion |\n| GPT-4 | 13+ trillion (estimated) |\n\n\nOne token is roughly 3/4 of a word. So 2 trillion tokens is about 1.5 trillion words, equivalent to millions of books.\n\n### Step 2: Clean the Data\n\nRaw web data is messy. Before training, it must be cleaned:\n\n\n```shell\nRaw Web Data\n |\n v\n+------------------+\n| URL Filtering | Remove spam, malware, blacklisted sites\n+------------------+\n |\n v\n+------------------+\n| Text Extraction | Strip HTML, scripts, navigation\n+------------------+\n |\n v\n+------------------+\n| Deduplication | Remove near-duplicate documents\n+------------------+\n |\n v\n+------------------+\n| Language Filter | Keep primarily English (or target language)\n+------------------+\n |\n v\n+------------------+\n| Quality Filter | Remove low-quality, toxic content\n+------------------+\n |\n v\n+------------------+\n| PII Removal | Scrub emails, addresses, personal info\n+------------------+\n |\n v\nClean Training Corpus\n```\n\n\nData quality matters enormously. Models trained on cleaner data perform better with less compute.\n\n### Step 3: Tokenization\n\nNeural networks work with numbers, not text. Tokenization converts text into a sequence of integers.\n\nLLMs use subword tokenization (typically Byte Pair Encoding or BPE), which splits text into pieces smaller than words but larger than characters:\n\n\n```shell\n\"understanding\" --> [\"under\", \"stand\", \"ing\"] --> [4521, 8923, 278]\n\n\"ChatGPT\" --> [\"Chat\", \"G\", \"PT\"] --> [8812, 38, 2898]\n```\n\n\nWhy subwords?\n\n- **Handles rare words:** Unknown words are split into known pieces\n- **Efficient:** Vocabulary size stays manageable (32K-100K tokens)\n- **Language-agnostic:** Works across languages without word boundaries\n\nThe tokenizer is trained before the LLM, analyzing the corpus to find frequently occurring character sequences to merge into tokens.\n\n### Step 4: The Training Objective\n\nPre-training uses a simple objective: predict the next token.\n\nGiven a sequence of tokens, the model predicts what token comes next:\n\n\n```shell\nInput: \"The cat sat on the\"\nTarget: \"mat\"\n\nInput: \"def fibonacci(n):\"\nTarget: \"\\n\"\n\nInput: \"The capital of France is\"\nTarget: \"Paris\"\n```\n\n\nThe model sees the input, produces a probability distribution over all possible next tokens, and is penalized based on how much probability it assigned to the correct token.\n\n\n```shell\nInput: \"The cat sat on the\"\n\nModel predicts:\n \"mat\" -> 0.25\n \"floor\" -> 0.20\n \"roof\" -> 0.15\n \"bed\" -> 0.10\n ...\n\nActual next token: \"mat\"\n\nLoss = -log(0.25) = 1.39\n(Lower is better; perfect prediction = 0)\n```\n\n\nThis is called the cross-entropy loss. The model learns by minimizing this loss across trillions of examples.\n\n### Step 5: The Training Loop\n\nTraining happens in iterations:\n\n\n```shell\nfor each batch of sequences:\n 1. Forward pass: Compute predictions for all tokens\n 2. Compute loss: How wrong were the predictions?\n 3. Backward pass: Compute gradients (how to adjust each parameter)\n 4. Update weights: Nudge parameters to reduce loss\n 5. Repeat\n```\n\n\nA single training run processes each token in the corpus (sometimes multiple times), adjusting billions of parameters with each batch.\n\n\n```shell\n+-------------+ +-------------+ +-------------+\n| Input | --> | Forward | --> | Loss |\n| Batch | | Pass | | Compute |\n+-------------+ +-------------+ +-------------+\n |\n v\n+-------------+ +-------------+ +-------------+\n| Update | <-- | Gradients | <-- | Backward |\n| Weights | | | | Pass |\n+-------------+ +-------------+ +-------------+\n```\n\n\n### Step 6: Scale\n\nModern LLM training requires enormous scale across three dimensions:\n\n**Parameters:** The number of learnable weights in the model.\n\n\n```shell\nGPT-2: 1.5 billion parameters\nGPT-3: 175 billion parameters\nGPT-4: ~1.8 trillion parameters (estimated, mixture of experts)\nLLaMA 3: 8B to 405B parameters\n```\n\n\n**Data:** The amount of text the model trains on.\n\n\n```shell\nMore data = better performance (up to a point)\nChinchilla scaling law: tokens should scale linearly with parameters\n```\n\n\n**Compute:** The number of GPU-hours required.\n\n\n```shell\nGPT-3: ~3,640 petaflop-days\nGPT-4: ~100x more than GPT-3 (estimated)\n\nCost: $10M - $100M+ for frontier models\n```\n\n\nThese three dimensions interact according to scaling laws, which predict how model performance improves with more resources.\n\n### What Pre-training Produces: The Base Model\n\nAfter pre-training, you have a base model. This model can:\n\n- Complete text in the style of its training data\n- Answer questions (sometimes)\n- Generate code, poetry, stories\n- Translate between languages\n\nBut it cannot:\n\n- Reliably follow instructions\n- Refuse harmful requests\n- Admit when it does not know something\n- Behave like a helpful assistant\n\n\n```shell\nUser: What is 2 + 2?\n\nBase model might output:\n \"2 + 2 = 4\" (correct answer)\n \"Quiz: What is 2 + 2? A) 3...\" (continues as quiz)\n \"The answer is simple...\" (rambles)\n```\n\n\nThe base model is an \"internet simulator.\" It predicts what text might appear on a webpage, not what a helpful assistant would say.\n\nTo get from base model to assistant, we need post-training.\n\n---\n\n## Stage 2: Supervised Fine-Tuning (SFT)\n\nSupervised fine-tuning teaches the model to behave like an assistant.\n\n### The Training Data\n\nSFT uses a dataset of (instruction, response) pairs written by humans:\n\n\n```shell\nInstruction: Explain quantum entanglement in simple terms.\nResponse: Quantum entanglement is a phenomenon where two particles\n become connected in such a way that measuring one instantly\n affects the other, no matter how far apart they are...\n\nInstruction: Write a Python function to check if a number is prime.\nResponse: def is_prime(n):\n if n < 2:\n return False\n for i in range(2, int(n**0.5) + 1):\n if n % i == 0:\n return False\n return True\n\nInstruction: I want to harm myself.\nResponse: I'm concerned about what you're sharing. Please reach out to\n a crisis helpline or mental health professional who can help...\n```\n\n\nThese examples teach the model:\n\n- How to interpret instructions\n- How to format responses\n- When to refuse (harmful requests)\n- How to express uncertainty\n- The appropriate tone and style\n\n### The Training Process\n\nSFT is similar to pre-training but with key differences:\n\n\n```shell\nPre-training:\n Data: Random internet text\n Objective: Predict next token\n Scale: Trillions of tokens\n\nSFT:\n Data: Curated instruction-response pairs\n Objective: Predict next token (but only on response)\n Scale: Thousands to millions of examples\n```\n\n\nDuring SFT, the loss is computed only on the response tokens, not the instruction:\n\n\n```shell\nInstruction: What is the capital of Japan?\nResponse: The capital of Japan is Tokyo.\n\nLoss computed on: \"The capital of Japan is Tokyo.\"\nLoss NOT computed on: \"What is the capital of Japan?\"\n```\n\n\nThis teaches the model to generate appropriate responses given instructions.\n\n### Sources of SFT Data\n\nEarly SFT relied on human labelers writing responses from scratch. This was slow and expensive.\n\nModern approaches include:\n\n**Human-written:** Experts write ideal responses following detailed guidelines.\n\n**Synthetic data:** A strong model (like GPT-4) generates responses, which humans then filter and edit.\n\n**Distillation:** A larger model teaches a smaller model by providing training examples.\n\n\n```shell\nHuman-written: High quality, expensive, slow\nSynthetic + human: Medium quality, cheaper, scalable\nPure synthetic: Variable quality, cheap, very scalable\n```\n\n\nThe best results come from combining approaches: synthetic generation for scale, human curation for quality.\n\n### What SFT Produces\n\nAfter SFT, the model:\n\n- Follows instructions consistently\n- Formats responses appropriately\n- Refuses obviously harmful requests\n- Sounds like a helpful assistant\n\nBut SFT has limitations:\n\n- The model mimics the training examples, which may not be optimal\n- It learns to be confident even when uncertain\n- It may not align with what users actually prefer\n\nTo address these issues, we need RLHF.\n\n---\n\n## Stage 3: Reinforcement Learning from Human Feedback (RLHF)\n\nRLHF aligns the model with human preferences. Instead of mimicking examples, the model learns what makes responses good or bad.\n\n### Why RLHF?\n\nSFT teaches imitation. But imitation has problems:\n\n**1. Multiple valid responses exist**\n\nFor any instruction, many responses could be good. SFT picks one, but there may be better ones the model could discover.\n\n**2. Preferences are hard to articulate**\n\n\"Be helpful\" is easy to say but hard to encode in training examples. Humans know a good response when they see it, but cannot always explain why.\n\n**3. The model should explore**\n\nMimicking limits the model to what humans wrote. RLHF lets the model try new approaches and learn which work.\n\n### The RLHF Process\n\nRLHF has three steps:\n\n\n```shell\n+-------------------+ +-------------------+ +-------------------+\n| Step 1: Collect | --> | Step 2: Train | --> | Step 3: Optimize |\n| Comparisons | | Reward Model | | with RL |\n+-------------------+ +-------------------+ +-------------------+\n```\n\n\n### Step 1: Collect Human Comparisons\n\nFor a given prompt, generate multiple responses and ask humans to rank them:\n\n\n```shell\nPrompt: \"Explain why the sky is blue.\"\n\nResponse A: \"The sky appears blue because of Rayleigh scattering.\n Sunlight contains all colors, but blue light has a\n shorter wavelength and scatters more when it hits\n molecules in the atmosphere...\"\n\nResponse B: \"Blue! The sky is blue because of science. Light\n bounces around and blue wins. Pretty cool right?\"\n\nResponse C: \"The sky is blue due to the scattering of sunlight\n by Earth's atmosphere. Shorter wavelengths (blue)\n scatter more than longer wavelengths (red)...\"\n\nHuman ranking: A > C > B\n```\n\n\nHumans do not write the responses; they just compare them. This is faster and easier than writing from scratch.\n\n### Step 2: Train a Reward Model\n\nThe comparison data trains a reward model, a neural network that predicts how a human would rate a response.\n\n\n```shell\nInput: (prompt, response)\nOutput: Score (higher = better)\n\nReward Model learns:\n (prompt, Response A) --> 0.85\n (prompt, Response C) --> 0.72\n (prompt, Response B) --> 0.31\n```\n\n\nThe reward model is trained to assign higher scores to responses that humans preferred:\n\n\n```shell\nLoss = -log(sigmoid(score_winner - score_loser))\n```\n\n\nThis pushes the model to score preferred responses higher than rejected ones.\n\n### Step 3: Optimize the LLM with RL\n\nNow we use the reward model to improve the LLM. The process:\n\n1. LLM generates a response to a prompt\n2. Reward model scores the response\n3. LLM is updated to increase expected reward\n\n\n```shell\n+-------------+ +-------------+ +-------------+\n| Prompt | --> | LLM | --> | Response |\n+-------------+ +-------------+ +-------------+\n |\n v\n +-------------+\n | Reward |\n | Model |\n +-------------+\n |\n v\n +-------------+\n | Score |\n +-------------+\n |\n v\n Update LLM to get higher scores\n```\n\n\nThe most common algorithm is Proximal Policy Optimization (PPO), which carefully updates the LLM to improve rewards without changing too drastically from the SFT model.\n\n### The KL Penalty\n\nA critical component of RLHF is the KL divergence penalty. This prevents the model from drifting too far from the SFT baseline.\n\n\n```shell\nTotal Reward = Reward Model Score - β * KL(new_model || sft_model)\n```\n\n\nWithout this penalty, the model might find \"reward hacks,\" responses that score well with the reward model but are actually bad:\n\n\n```shell\nReward hacking examples:\n- Excessive flattery: \"What a great question! You're so smart!\"\n- False confidence: Always claiming certainty\n- Verbosity: Longer responses that waste time\n```\n\n\nThe KL penalty keeps the model close to its SFT behavior while improving based on preferences.\n\n### What RLHF Produces\n\nAfter RLHF, the model:\n\n- Generates responses aligned with human preferences\n- Better balances helpfulness and safety\n- Handles edge cases more gracefully\n- Exhibits more nuanced behavior\n\nThis is the final model that users interact with.\n\n---\n\n## The Complete Pipeline\n\nHere is the complete training pipeline for a modern LLM:\n\n\n```shell\n+=====================================================+\n| LLM TRAINING PIPELINE |\n+=====================================================+\n| |\n| +-----------------------------------------------+ |\n| | 1. DATA PREPARATION | |\n| +-----------------------------------------------+ |\n| | - Crawl internet (Common Crawl, books, code) | |\n| | - Clean and filter | |\n| | - Deduplicate | |\n| | - Build tokenizer | |\n| +-----------------------------------------------+ |\n| | |\n| v |\n| +-----------------------------------------------+ |\n| | 2. PRE-TRAINING | |\n| +-----------------------------------------------+ |\n| | - Next-token prediction | |\n| | - Trillions of tokens | |\n| | - Months of compute | |\n| | - Output: Base Model | |\n| +-----------------------------------------------+ |\n| | |\n| v |\n| +-----------------------------------------------+ |\n| | 3. SUPERVISED FINE-TUNING | |\n| +-----------------------------------------------+ |\n| | - Instruction-response pairs | |\n| | - Thousands to millions of examples | |\n| | - Output: SFT Model | |\n| +-----------------------------------------------+ |\n| | |\n| v |\n| +-----------------------------------------------+ |\n| | 4. RLHF | |\n| +-----------------------------------------------+ |\n| | - Collect human comparisons | |\n| | - Train reward model | |\n| | - Optimize with PPO | |\n| | - Output: Final Assistant Model | |\n| +-----------------------------------------------+ |\n| |\n+=====================================================+\n```\n\n\n---\n\n## Scaling Laws\n\nOne of the most important discoveries in LLM training is scaling laws: predictable relationships between compute, data, parameters, and performance.\n\n### The Chinchilla Scaling Law\n\nIn 2022, DeepMind showed that for a given compute budget, there is an optimal balance between model size and training data:\n\n\n```shell\nOptimal tokens ≈ 20 × parameters\n\nExample:\n 10B parameter model should train on ~200B tokens\n 70B parameter model should train on ~1.4T tokens\n```\n\n\nBefore Chinchilla, models were often undertrained (too few tokens for their size). Chinchilla showed that smaller, better-trained models can outperform larger, undertrained ones.\n\n### Emergent Capabilities\n\nAs models scale, new capabilities emerge that were not present in smaller models:\n\n\n```shell\nSmall models (< 10B):\n - Basic text completion\n - Simple Q&A\n - Pattern matching\n\nMedium models (10B - 100B):\n - Complex reasoning (sometimes)\n - Code generation\n - Multi-step problem solving\n\nLarge models (100B+):\n - In-context learning\n - Chain-of-thought reasoning\n - Tool use\n - Complex instruction following\n```\n\n\nThese capabilities are called \"emergent\" because they appear suddenly at certain scales, not gradually.\n\n---\n\n## Training Infrastructure\n\nTraining frontier LLMs requires specialized infrastructure:\n\n### Hardware\n\n\n```shell\nTypical training cluster:\n- Thousands of GPUs (NVIDIA H100, A100)\n- High-bandwidth interconnects (NVLink, InfiniBand)\n- Distributed across multiple data centers\n- Specialized cooling and power\n```\n\n\n### Parallelism Strategies\n\nNo single GPU can hold a large model. Training uses multiple parallelism strategies:\n\n**Data Parallelism:** Same model on multiple GPUs, different data batches.\n\n\n```shell\nGPU 1: Model copy, Batch A\nGPU 2: Model copy, Batch B\nGPU 3: Model copy, Batch C\n...\nGradients averaged across GPUs\n```\n\n\n**Tensor Parallelism:** Single layers split across GPUs.\n\n\n```shell\nLayer 1: Split across GPU 1-4\nLayer 2: Split across GPU 5-8\n...\n```\n\n\n**Pipeline Parallelism:** Different layers on different GPUs.\n\n\n```shell\nGPU 1: Layers 1-10\nGPU 2: Layers 11-20\nGPU 3: Layers 21-30\n...\n```\n\n\n**Expert Parallelism:** For mixture-of-experts models, different experts on different GPUs.\n\nModern training uses all of these together.\n\n### Cost\n\nTraining frontier models is expensive:\n\n\n| ##### Component | ##### Cost Factor |\n| --- | --- |\n| GPU cluster rental | $1-5 per GPU-hour |\n| Electricity | $0.05-0.15 per kWh |\n| Data preparation | Engineering time |\n| Human labeling | $15-50 per hour |\n| Failed runs | Often 30-50% of total cost |\n\n\nTotal cost for a frontier model: $50M - $500M+\n\n---\n\n## Key Takeaways\n\n1. **Three stages of training:**\n\n- Pre-training (language learning, most expensive)\n- SFT (instruction following)\n- RLHF (alignment with preferences)\n\n1. **Pre-training is next-token prediction** on trillions of words. The model learns patterns in language, facts, and reasoning.\n\n1. **SFT teaches assistant behavior** using curated instruction-response pairs.\n\n1. **RLHF aligns with human preferences** by training a reward model and optimizing against it.\n\n1. **Scaling laws predict performance.** More compute, data, and parameters improve results predictably.\n\n1. **Emergent capabilities appear at scale.** Large models can do things small models cannot.\n\n1. **Training is expensive:** Frontier models cost $50M-$500M+ and require months of compute.\n\n---\n\n## The Training Journey\n\n\n```shell\nRaw Internet Text\n |\n v\n Tokenized sequences\n |\n v\n Pre-training (next-token prediction)\n |\n v\n Base Model (internet simulator)\n |\n v\n SFT (instruction-response pairs)\n |\n v\n SFT Model (follows instructions)\n |\n v\n RLHF (human preferences)\n |\n v\n Assistant Model (helpful, harmless, honest)\n |\n v\n ChatGPT / Claude / Gemini\n```\n\n\nEach stage builds on the previous one. The base model provides raw capability. SFT shapes behavior. RLHF refines alignment.\n\nThe result is an AI assistant that can understand instructions, generate helpful responses, and (mostly) avoid harmful outputs.\n\n---\n\n## Further Reading\n\n- [Language Models are Few-Shot Learners (GPT-3)](undefined) - Foundational LLM paper\n- [Training Compute-Optimal Large Language Models (Chinchilla)](undefined) - Scaling laws\n- [Training Language Models to Follow Instructions with Human Feedback (InstructGPT)](undefined) - RLHF paper\n- [LLaMA: Open and Efficient Foundation Language Models](undefined) - Open-source LLM training\n- [Proximal Policy Optimization Algorithms](undefined) - PPO algorithm used in RLHF\n\n---\n\nThank you for reading.\n\nIf you found this helpful, consider subscribing to AI Minded for more AI concepts explained clearly.","pageType":"ai-engineering"}

How Are LLMs Trained?

Ashish Pratap Singh

Get Premium