{"title":"Diffusion Models Explained","description":"","content":"How do you turn random static into a photorealistic image? It sounds impossible, but that's exactly what diffusion models do.\n\nThe key insight is counterintuitive: instead of learning to generate images directly, diffusion models learn to remove noise. Train a model to clean up slightly corrupted images, then apply that cleaning process repeatedly to pure noise. After enough iterations, coherent images emerge from chaos.\n\nThis simple idea powers Stable Diffusion, DALL-E, Midjourney, and Sora. Diffusion models have become the dominant approach for image, video, and audio generation. They produce higher quality outputs than GANs, are easier to train, and offer more control over the generation process.\n\nIn this chapter, I'll explain how diffusion models work, the math behind them, and why they've become so successful.\n\n---\n\n# What is a Diffusion Model?\n\nA diffusion model is a generative model that learns to reverse a gradual noising process. It has two phases:\n\n1. **Forward process (diffusion)**: Gradually add noise to data until it becomes pure random noise\n2. **Reverse process (denoising)**: Learn to remove noise step by step, recovering the original data\n\n\n```shell\nForward Process (Training):\n+----------+ +----------+ +----------+ +----------+ +----------+\n| Clean | -> | Slightly | -> | Noisy | -> | Very | -> | Pure |\n| Image | | Noisy | | Image | | Noisy | | Noise |\n+----------+ +----------+ +----------+ +----------+ +----------+\n t = 0 t = 250 t = 500 t = 750 t = 1000\n\nReverse Process (Generation):\n+----------+ +----------+ +----------+ +----------+ +----------+\n| Pure | -> | Less | -> | Shapes | -> | Clear | -> | Generated|\n| Noise | | Noisy | | Emerge | | Image | | Image |\n+----------+ +----------+ +----------+ +----------+ +----------+\n t = 1000 t = 750 t = 500 t = 250 t = 0\n```\n\n\n**The analogy**: Imagine dropping ink into water. The ink slowly diffuses until it's evenly spread throughout. If you filmed this and played it backward, you'd see the ink reconcentrate into a drop. Diffusion models learn to \"play the video backward,\" turning uniform noise into structured data.\n\nThe model never sees pure noise during training. It only learns to remove small amounts of noise at each step. But by chaining these small steps together, it can denoise from any noise level, including pure random noise.\n\n---\n\n# Why Diffusion Works Better Than GANs\n\nGANs were the previous state-of-the-art for image generation. They train two networks in competition: a generator creates fakes, and a discriminator detects them. This adversarial setup is powerful but has problems.\n\nDiffusion models avoid these issues:\n\n\n| ##### Challenge | ##### GANs | ##### Diffusion Models |\n| --- | --- | --- |\n| **Training stability** | Unstable, requires careful balancing | Stable, just minimize MSE loss |\n| **Mode collapse** | Generator may only produce few variations | Naturally covers full distribution |\n| **Quality vs diversity** | Often trade one for the other | Both high quality and diverse |\n| **Evaluation** | Hard to measure progress during training | Loss directly indicates quality |\n\n\nThe key difference is in the training objective.\n\n#### GAN Training:\n\n- Generator: Fool the discriminator\n- Discriminator: Catch fakes\n- Two competing objectives that can destabilize\n\n#### Diffusion Training:\n\n- Single network: Predict the noise that was added\n- One simple objective: Minimize prediction error\n- Stable gradient flow throughout\n\nDiffusion models also have a theoretical advantage. The reverse process directly models the data distribution, so they can capture all the modes (variations) in the training data. GANs can miss modes because the discriminator might not notice if some variations are missing.\n\n---\n\n# The Forward Process: Adding Noise\n\nThe forward process is fixed (not learned). It gradually corrupts an image by adding Gaussian noise according to a schedule.\n\nAt each timestep t, we add a small amount of noise:\n\n\n```shell\nx_t = sqrt(alpha_t) * x_{t-1} + sqrt(1 - alpha_t) * noise\n\nWhere:\n- x_t is the image at timestep t\n- x_{t-1} is the image at the previous timestep\n- alpha_t controls how much signal vs noise (close to 1)\n- noise is sampled from N(0, I)\n```\n\n\nInstead of computing this step by step, we can jump directly to any timestep:\n\n\n```shell\nx_t = sqrt(alpha_bar_t) * x_0 + sqrt(1 - alpha_bar_t) * noise\n\nWhere:\n- alpha_bar_t = alpha_1 * alpha_2 * ... * alpha_t (cumulative product)\n- x_0 is the original clean image\n```\n\n\nThis is crucial for training efficiency. We don't need to simulate 1000 steps. We can directly compute what the image looks like at any noise level.\n\n\n```shell\nNoise Schedule Example (1000 timesteps):\n\nTimestep: 0 250 500 750 1000\nalpha_bar: 1.0 0.75 0.50 0.25 0.00\n\nSignal: 100% 86% 70% 50% 0%\nNoise: 0% 14% 30% 50% 100%\n\n+----------+----------+----------+----------+----------+\n| Clear | Slight | Noisy | Very | Pure |\n| Image | Haze | Image | Noisy | Noise |\n+----------+----------+----------+----------+----------+\n```\n\n\nThe schedule determines how quickly noise is added. Linear schedules (used in early papers) add noise at a constant rate. Cosine schedules (used in improved models) add noise more slowly at the start and end, which improves quality.\n\n---\n\n# The Reverse Process: Learning to Denoise\n\nThe reverse process is where learning happens. We train a neural network to predict the noise in a noisy image.\n\n\n```shell\nTraining Loop:\n+-----------------------------------------------------------+\n| 1. Sample clean image x_0 from dataset |\n| 2. Sample random timestep t from [1, 1000] |\n| 3. Sample random noise epsilon ~ N(0, I) |\n| 4. Create noisy image: x_t = sqrt(a_t)*x_0 + sqrt(1-a_t)*eps |\n| 5. Network predicts noise: eps_pred = model(x_t, t) |\n| 6. Loss = MSE(eps_pred, epsilon) |\n| 7. Update model weights via gradient descent |\n+-----------------------------------------------------------+\n```\n\n\nThe model takes two inputs:\n\n1. **Noisy image x_t**: The corrupted image\n2. **Timestep t**: How much noise was added (tells the model what level of denoising to do)\n\nAnd outputs:\n\n- **Predicted noise**: What noise the model thinks is in the image\n\n NET\\n T --> NET\\n NET --> EPS\\n\\n classDef primary fill:#00ceff,stroke:#0099cc,color:#000\\n classDef orange fill:#ffa94d,stroke:#e09030,color:#000\\n classDef green fill:#69db7c,stroke:#4fc463,color:#000\\n classDef purple fill:#9775fa,stroke:#7c5ce0,color:#000\"} -->\n```mermaid\nflowchart TD\n XT[\"Noisy Image (x_t)\"]:::primary\n T[\"Timestep (t)\"]:::orange\n\n subgraph NET[\"Denoising Network\"]\n UNET[\"U-Net Architecture\"]:::purple\n end\n\n EPS[\"Predicted Noise (ε_pred)\"]:::green\n\n XT --> NET\n T --> NET\n NET --> EPS\n\n classDef primary fill:#00ceff,stroke:#0099cc,color:#000\n classDef orange fill:#ffa94d,stroke:#e09030,color:#000\n classDef green fill:#69db7c,stroke:#4fc463,color:#000\n classDef purple fill:#9775fa,stroke:#7c5ce0,color:#000\n```\n\n\nTraining is remarkably simple: just minimize the mean squared error between predicted noise and actual noise. No adversarial dynamics, no mode collapse, no instability.\n\n---\n\n# Generation: Iterative Denoising\n\nTo generate an image, we start with pure random noise and repeatedly denoise:\n\n\n```shell\nGeneration Process:\n+-----------------------------------------------------------+\n| 1. Sample pure noise: x_T ~ N(0, I) |\n| 2. For t = T, T-1, T-2, ..., 1: |\n| a. Predict noise: eps = model(x_t, t) |\n| b. Remove predicted noise to get x_{t-1} |\n| c. Add small random noise (except at final step) |\n| 3. Return x_0 as the generated image |\n+-----------------------------------------------------------+\n```\n\n\nThe denoising step is:\n\n\n```shell\nx_{t-1} = (1/sqrt(alpha_t)) * (x_t - (1-alpha_t)/sqrt(1-alpha_bar_t) * eps_pred) + sigma_t * z\n\nWhere:\n- First part removes the predicted noise\n- sigma_t * z adds small random noise for stochasticity\n- z ~ N(0, I) for t > 1, z = 0 for t = 1\n```\n\n\nThe added noise during generation isn't a bug. It's a feature. It makes the process stochastic, so different random seeds produce different outputs even with the same starting noise.\n\n\n```shell\nStep-by-step Example (simplified):\n\nt=1000: x_1000 = pure random noise\n |\n v (predict noise, remove it)\nt=800: x_800 = less noisy, vague shapes emerging\n |\n v\nt=600: x_600 = general structure visible\n |\n v\nt=400: x_400 = clear composition, details fuzzy\n |\n v\nt=200: x_200 = most details present\n |\n v\nt=0: x_0 = final generated image\n```\n\n\n---\n\n# The Network Architecture: U-Net\n\nMost diffusion models use a U-Net architecture. The U-Net has an encoder-decoder structure with skip connections:\n\n DOWN1\\n TIME --> DOWN1\\n TIME --> DOWN2\\n TIME --> DOWN3\\n TIME --> BOTTLENECK\\n TIME --> UP1\\n TIME --> UP2\\n TIME --> UP3\\n\\n DOWN1 --> DOWN2 --> DOWN3 --> BOTTLENECK\\n BOTTLENECK --> UP1 --> UP2 --> UP3 --> OUTPUT\\n\\n DOWN3 -.->|skip| UP1\\n DOWN2 -.->|skip| UP2\\n DOWN1 -.->|skip| UP3\\n\\n classDef primary fill:#00ceff,stroke:#0099cc,color:#000\\n classDef orange fill:#ffa94d,stroke:#e09030,color:#000\\n classDef teal fill:#38d9a9,stroke:#2eb88a,color:#000\\n classDef green fill:#69db7c,stroke:#4fc463,color:#000\\n classDef purple fill:#9775fa,stroke:#7c5ce0,color:#000\"} -->\n```mermaid\nflowchart TD\n INPUT[\"Input x_t\"]:::primary\n TIME[\"Timestep
Embedding\"]:::orange\n OUTPUT[\"Output ε_pred\"]:::green\n\n subgraph ENCODER[\"Encoder (Downsampling)\"]\n DOWN1[\"Down Block
64x64\"]:::teal\n DOWN2[\"Down Block
32x32\"]:::teal\n DOWN3[\"Down Block
16x16\"]:::teal\n end\n\n BOTTLENECK[\"Bottleneck
8x8\"]:::purple\n\n subgraph DECODER[\"Decoder (Upsampling)\"]\n UP1[\"Up Block
16x16\"]:::orange\n UP2[\"Up Block
32x32\"]:::orange\n UP3[\"Up Block
64x64\"]:::orange\n end\n\n INPUT --> DOWN1\n TIME --> DOWN1\n TIME --> DOWN2\n TIME --> DOWN3\n TIME --> BOTTLENECK\n TIME --> UP1\n TIME --> UP2\n TIME --> UP3\n\n DOWN1 --> DOWN2 --> DOWN3 --> BOTTLENECK\n BOTTLENECK --> UP1 --> UP2 --> UP3 --> OUTPUT\n\n DOWN3 -.->|skip| UP1\n DOWN2 -.->|skip| UP2\n DOWN1 -.->|skip| UP3\n\n classDef primary fill:#00ceff,stroke:#0099cc,color:#000\n classDef orange fill:#ffa94d,stroke:#e09030,color:#000\n classDef teal fill:#38d9a9,stroke:#2eb88a,color:#000\n classDef green fill:#69db7c,stroke:#4fc463,color:#000\n classDef purple fill:#9775fa,stroke:#7c5ce0,color:#000\n```\n\n\n### Key components:\n\n#### Downsampling Blocks\n\nExtract features at progressively lower resolutions. Each block typically contains:\n\n- Convolutional layers\n- Normalization (group norm or layer norm)\n- Activation (SiLU/Swish is common)\n- Downsampling (strided conv or pooling)\n\n#### Upsampling Blocks\n\nReconstruct the output at increasing resolutions:\n\n- Transposed convolutions or nearest-neighbor upsampling\n- Concatenation with skip connection from encoder\n- Same conv, norm, activation pattern\n\n#### Skip Connections\n\nDirectly connect encoder layers to corresponding decoder layers. This preserves fine details that might be lost during downsampling.\n\n#### Timestep Embedding\n\nThe timestep t is embedded into a vector and injected into every block:\n\n\n```shell\nt (scalar) -> Sinusoidal Encoding -> MLP -> Timestep Embedding\n\nThe embedding is added to or modulates each residual block,\ntelling the network what noise level to expect.\n```\n\n\n#### Attention Layers\n\nSelf-attention layers (often at lower resolutions) help the model capture global relationships. For text-to-image models, cross-attention layers attend to text embeddings.\n\n---\n\n# Making It Faster: DDIM and Other Samplers\n\nThe original diffusion process (DDPM) requires hundreds or thousands of denoising steps. This is slow. A 50-step generation takes several seconds on a GPU.\n\n**DDIM (Denoising Diffusion Implicit Models)** changes the denoising equation to be deterministic.\n\n#### DDPM (stochastic)\n\n- Each step adds random noise\n- Must take many small steps\n- Different seeds give different results even mid-generation\n\n#### DDIM (deterministic)\n\n- No random noise during generation\n- Can skip steps (e.g., only use 50 of 1000 timesteps)\n- Same starting noise always gives same output\n\nDDIM enables a quality vs speed tradeoff:\n\n\n| ##### Steps | ##### Quality | ##### Time |\n| --- | --- | --- |\n| 1000 | Best | Slow |\n| 50 | Very good | Reasonable |\n| 20 | Good | Fast |\n| 10 | Acceptable | Very fast |\n\n\nOther popular samplers:\n\n- **DPM-Solver**: Uses differential equation solvers for even fewer steps\n- **Euler**: Simple and fast, good at low step counts\n- **Heun**: Higher order solver, more accurate per step\n\n---\n\n# Latent Diffusion: Making It Efficient\n\nRunning diffusion on full-resolution images is computationally expensive. A 512x512 image has 786,432 pixels to process at every step.\n\n**Latent Diffusion Models** (like Stable Diffusion) solve this by working in a compressed space:\n\n\n```shell\nStandard Diffusion:\n512x512x3 image -> Denoise at full resolution -> 512x512x3 output\n(786,432 values per step)\n\nLatent Diffusion:\n512x512x3 image -> VAE Encoder -> 64x64x4 latent\n |\n v\n Denoise in latent space\n (16,384 values per step)\n |\n v\n VAE Decoder -> 512x512x3 output\n```\n\n\nThis is 48x less computation. The VAE (Variational Autoencoder) learns to compress images into a lower-dimensional space that preserves semantic information:\n\n\n```shell\n+-------------+ +-------------+ +-------------+\n| Image | | Latent | | Reconstructed|\n| 512x512x3 | -----> | 64x64x4 | -----> | 512x512x3 |\n| | Encode | | Decode | |\n+-------------+ +-------------+ +-------------+\n```\n\n\nThe VAE is trained separately to minimize reconstruction error. Once trained, its weights are frozen. The diffusion model only operates in latent space, never touching raw pixels during training or generation.\n\n---\n\n# Conditioning: Guiding Generation\n\nUnconditional diffusion models generate random samples from the training distribution. For practical use, we need control over what gets generated.\n\n### Class Conditioning\n\nThe simplest form. Add a class label embedding to the timestep embedding:\n\n NET\\n T --> NET\\n CLASS --> EMB --> NET\\n NET --> OUT\\n\\n classDef primary fill:#00ceff,stroke:#0099cc,color:#000\\n classDef orange fill:#ffa94d,stroke:#e09030,color:#000\\n classDef teal fill:#38d9a9,stroke:#2eb88a,color:#000\\n classDef green fill:#69db7c,stroke:#4fc463,color:#000\\n classDef purple fill:#9775fa,stroke:#7c5ce0,color:#000\"} -->\n```mermaid\nflowchart LR\n XT[\"x_t\"]:::primary\n T[\"t\"]:::orange\n CLASS[\"class_label
= 'cat'\"]:::teal\n EMB[\"embedding\"]:::purple\n NET[\"Network\"]:::orange\n OUT[\"Output\"]:::green\n\n XT --> NET\n T --> NET\n CLASS --> EMB --> NET\n NET --> OUT\n\n classDef primary fill:#00ceff,stroke:#0099cc,color:#000\n classDef orange fill:#ffa94d,stroke:#e09030,color:#000\n classDef teal fill:#38d9a9,stroke:#2eb88a,color:#000\n classDef green fill:#69db7c,stroke:#4fc463,color:#000\n classDef purple fill:#9775fa,stroke:#7c5ce0,color:#000\n```\n\n\nThe model learns to generate images matching the given class.\n\n### Text Conditioning (CLIP)\n\nModern text-to-image models use CLIP to encode text prompts:\n\n CLIP --> EMB --> CROSS --> OUTPUT\\n\\n classDef primary fill:#00ceff,stroke:#0099cc,color:#000\\n classDef orange fill:#ffa94d,stroke:#e09030,color:#000\\n classDef teal fill:#38d9a9,stroke:#2eb88a,color:#000\\n classDef green fill:#69db7c,stroke:#4fc463,color:#000\\n classDef purple fill:#9775fa,stroke:#7c5ce0,color:#000\"} -->\n```mermaid\nflowchart TD\n PROMPT[\"'A cat wearing a hat'\"]:::primary\n CLIP[\"CLIP Text Encoder\"]:::orange\n EMB[\"Text Embedding
[77 x 768]\"]:::teal\n CROSS[\"Cross-attention
in U-Net\"]:::purple\n OUTPUT[\"Generated Image\"]:::green\n\n PROMPT --> CLIP --> EMB --> CROSS --> OUTPUT\n\n classDef primary fill:#00ceff,stroke:#0099cc,color:#000\n classDef orange fill:#ffa94d,stroke:#e09030,color:#000\n classDef teal fill:#38d9a9,stroke:#2eb88a,color:#000\n classDef green fill:#69db7c,stroke:#4fc463,color:#000\n classDef purple fill:#9775fa,stroke:#7c5ce0,color:#000\n```\n\n\nEach layer in the U-Net has cross-attention where:\n\n- **Query**: comes from image features\n- **Key/Value**: come from text embedding\n\nThis lets different spatial locations attend to relevant words in the prompt.\n\n### Classifier-Free Guidance\n\nMakes the model follow the conditioning more strongly:\n\n`eps_guided = eps_uncond + guidance_scale * (eps_cond - eps_uncond)`\n\n##### Where:\n\n- **eps_cond:** noise prediction with text conditioning\n- **eps_uncond:** noise prediction without conditioning (empty text)\n- **guidance_scale:** typically 7-10\n\nHigher guidance scale means stronger adherence to the prompt but can reduce diversity and introduce artifacts.\n\n---\n\n# Types of Diffusion Models\n\n### DDPM (Denoising Diffusion Probabilistic Models)\n\nThe original formulation. Trains by predicting noise, generates through iterative denoising. Requires many steps for good quality.\n\n### DDIM (Denoising Diffusion Implicit Models)\n\nDeterministic variant that allows fewer generation steps. Same training, different sampling.\n\n### Score-Based Models\n\nInstead of predicting noise, these predict the \"score\" (gradient of log probability). Mathematically equivalent to noise prediction but provides different intuition.\n\n### Flow Matching\n\nRecent alternative that learns to transform noise to data through continuous flows. Can be faster and more stable than standard diffusion.\n\n### Consistency Models\n\nDistill a diffusion model into one that can generate in a single step. Trade some quality for massive speed improvement.\n\n---\n\n# Applications Beyond Images\n\nDiffusion models aren't limited to images:\n\n### Video Generation (Sora, Runway)\n\nExtend to 3D tensors (width x height x time). Use temporal attention layers for consistency across frames.\n\n\n```shell\nImage: [H, W, C]\nVideo: [T, H, W, C] or [H, W, T, C]\n\nAdditional architectural needs:\n- Temporal attention (attend across frames)\n- Temporal convolutions (3D convs)\n- Frame-to-frame consistency mechanisms\n```\n\n\n### Audio Generation (AudioLDM, Riffusion)\n\nConvert audio to spectrograms (2D images) and apply image diffusion, or work directly on waveforms.\n\n### 3D Generation (Point-E, DreamFusion)\n\nGenerate 3D point clouds, meshes, or neural radiance fields.\n\n### Molecule Generation\n\nGenerate molecular structures for drug discovery. The \"image\" is a molecular graph.\n\n### Text Generation\n\nApply diffusion to text embeddings. Less common than autoregressive models but an active research area.\n\n---\n\n# Limitations and Challenges\n\n### Slow Generation\n\nEven with optimizations, diffusion models are slower than GANs or VAEs. A single forward pass vs 20+ denoising steps.\n\n### Memory Requirements\n\nLarge U-Net models with attention require significant GPU memory. Stable Diffusion XL needs 6+ GB VRAM.\n\n### Fine-Grained Control\n\nWhile guidance helps, precise control over specific image regions or attributes remains challenging.\n\n### Text in Images\n\nDiffusion models struggle to render readable text. The generation process operates on pixels, not glyphs.\n\n### Hands and Anatomy\n\nDetailed human anatomy, especially hands, often has errors. The model lacks explicit structural understanding.\n\n### Consistency\n\nGenerating the same character or object consistently across multiple images requires techniques like IP-Adapter or fine-tuning.\n\n---\n\n# Key Takeaways\n\n1. **Diffusion models learn to reverse a noising process.** Add noise gradually during training, remove it gradually during generation.\n2. **Training is simple and stable.** Just predict the noise that was added. No adversarial dynamics or mode collapse.\n3. **Generation is iterative.** Start with pure noise, denoise step by step. More steps generally means higher quality.\n4. **Latent diffusion makes it efficient.** Working in compressed space reduces computation by 48x while maintaining quality.\n5. **Conditioning enables control.** Class labels, text embeddings, and classifier-free guidance let you specify what to generate.\n6. **Faster samplers reduce steps.** DDIM and DPM-Solver generate good images in 20-50 steps instead of 1000.\n7. **The approach generalizes beyond images.** Video, audio, 3D, and molecules all work with diffusion frameworks.\n\n---\n\n# References\n\n- [Denoising Diffusion Probabilistic Models](undefined) - The foundational DDPM paper by Ho et al.\n- [Denoising Diffusion Implicit Models](undefined) - DDIM paper enabling faster sampling\n- [High-Resolution Image Synthesis with Latent Diffusion Models](undefined) - Stable Diffusion paper\n- [Classifier-Free Diffusion Guidance](undefined) - How text guidance works\n- [The Illustrated Stable Diffusion](undefined) - Visual explanation by Jay Alammar\n- [What are Diffusion Models?](undefined) - Comprehensive technical overview by Lilian Weng","pageType":"ai-engineering"}

Diffusion Models Explained

Ashish Pratap Singh

Get Premium