{"title":"11.8 Quantization","description":"","content":"# 11.8 Quantization\n\n## The Problem\n\nA 70B parameter model stored in FP16 (half precision) requires 140GB of VRAM just for the weights. A single A100 GPU, one of the most powerful cards available, has 80GB. So you need at least two A100s, costing $5-8 per hour in the cloud, just to load the model before you process a single request.\n\nThis is the fundamental tension in deploying large language models. The models that produce the best results are often too large to fit on the hardware you can afford. You could use a smaller model, but then you sacrifice quality. You could rent more GPUs, but that blows your budget. There has to be a middle ground.\n\nThat middle ground is quantization. Instead of storing each weight as a 16-bit floating point number, you store it as an 8-bit integer, a 4-bit integer, or even lower. A 70B model in FP16 takes 140GB. The same model in INT4 takes about 35GB, fitting comfortably on a single A100. The cost of running it drops by half or more, the inference speed improves because there is less data to move through memory, and the quality loss is often surprisingly small.\n\nQuantization is not a new idea. It has been used in signal processing and image compression for decades. But applying it to LLMs required solving some unique challenges, because not all weights in a neural network are equally important, and naive quantization can destroy the model's ability to reason. The techniques we will cover in this chapter are purpose-built for LLMs and represent the current state of the art.\n\n**What you will learn:**\n\n- How number formats (FP32, FP16, BF16, INT8, INT4) determine memory usage and precision\n- The mechanics of quantization: mapping high-precision values to low-precision representations\n- Post-Training Quantization (PTQ): GPTQ, AWQ, and GGUF formats and when to use each\n- Quantization-Aware Training (QAT) and when the extra effort pays off\n- Which tasks are robust to quantization and which are sensitive\n- A practical decision framework for choosing quantization formats\n- Hands-on code for quantizing and loading quantized models\n\n---\n\n## Number Formats Explained\n\nBefore diving into quantization techniques, you need to understand what you are actually changing. Every parameter in a neural network is stored as a number. The format of that number determines how much memory it uses and how precisely it can represent values.\n\n### FP32 (Full Precision)\n\nFP32 uses 32 bits (4 bytes) per parameter. It has 1 sign bit, 8 exponent bits, and 23 mantissa bits. This gives it the ability to represent roughly 4.2 billion distinct values across a massive range (approximately 1.18 x 10^-38 to 3.4 x 10^38). FP32 was the default for training neural networks for years, and it still is for many training workloads. But for inference, it is overkill. Most model weights cluster in a narrow range (typically between -1 and 1), so all that dynamic range goes to waste.\n\n### FP16 and BF16 (Half Precision)\n\nFP16 uses 16 bits (2 bytes) per parameter. It has 1 sign bit, 5 exponent bits, and 10 mantissa bits. It can represent about 65,536 distinct values. That sounds like a dramatic drop from FP32's 4.2 billion, but in practice it is plenty for storing trained model weights. FP16 is the de facto standard for LLM inference today.\n\nBF16 (Brain Float 16) is a variant that uses the same 16 bits but allocates them differently: 1 sign bit, 8 exponent bits, and only 7 mantissa bits. BF16 has the same range as FP32 (same number of exponent bits) but less precision. Google developed it specifically for deep learning, where having a wide range matters more than having ultra-fine precision. Most modern GPUs (A100, H100, RTX 4090) have dedicated BF16 compute units.\n\n### INT8 (8-bit Integer)\n\nINT8 uses 8 bits (1 byte) per parameter and can represent exactly 256 distinct values (0-255 for unsigned, or -128 to 127 for signed). Compared to FP16's 65,536 values, you are working with a much coarser grid. But here is the thing: neural network weights are not uniformly distributed. They tend to cluster around zero with a roughly bell-shaped distribution. A well-designed quantization scheme maps those 256 values to the ranges where most weights actually live, and the approximation error ends up being remarkably small.\n\n### INT4 (4-bit Integer)\n\nINT4 uses 4 bits (0.5 bytes) per parameter and can represent only 16 distinct values. This is where things get aggressive. Sixteen values to represent the full range of a weight that could originally take on 4.2 billion distinct values. And yet, with the right techniques, INT4 quantized models can retain 95-99% of their original quality. We will see how this is possible when we cover AWQ and GPTQ later in this chapter.\n\n### Memory Footprint Comparison\n\nHere is what these number formats mean in practice for a 7B parameter model.\n\n\n```mermaid\nflowchart LR\n subgraph FP32[\"FP32 (32-bit)\"]\n FP32_SIZE[\"28 GB\"]:::red\n end\n\n subgraph FP16[\"FP16 (16-bit)\"]\n FP16_SIZE[\"14 GB\"]:::orange\n end\n\n subgraph INT8[\"INT8 (8-bit)\"]\n INT8_SIZE[\"7 GB\"]:::primary\n end\n\n subgraph INT4[\"INT4 (4-bit)\"]\n INT4_SIZE[\"3.5 GB\"]:::green\n end\n\n FP32 --> FP16 --> INT8 --> INT4\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nEach step halves the memory. Going from FP32 to INT4 is an 8x reduction. A model that needed a multi-GPU server now fits on a laptop.\n\n### Format Comparison Table\n\n\n| Format | Bits | Bytes per Param | Distinct Values | Dynamic Range | Typical Use |\n|--------|------|-----------------|-----------------|---------------|-------------|\n| FP32 | 32 | 4 | ~4.2 billion | Very wide | Training (legacy) |\n| FP16 | 16 | 2 | ~65,536 | Moderate | Default inference |\n| BF16 | 16 | 2 | ~65,536 | Wide (same as FP32) | Training + inference on modern GPUs |\n| INT8 | 8 | 1 | 256 | Fixed (requires scaling) | Quality-sensitive quantized inference |\n| INT4 | 4 | 0.5 | 16 | Very narrow (requires scaling) | Memory-constrained deployment |\n\n\nThe \"requires scaling\" note for INT8 and INT4 is important. Integer formats do not have an exponent, so they cannot natively represent the range of floating-point values. Quantization techniques add a scale factor and sometimes a zero-point offset to map integer values back to the original floating-point range. That is exactly what we cover next.\n\n---\n\n## How Quantization Works\n\nThe core idea behind quantization is straightforward: take a set of high-precision floating-point values and map them to a smaller set of low-precision values. The goal is to do this in a way that preserves as much information as possible.\n\n### Symmetric Quantization\n\nThe simplest approach is symmetric quantization. You find the maximum absolute value in your weight tensor, compute a scale factor, and then round every weight to the nearest integer in the target range.\n\nThe math looks like this:\n\n- **Scale:** `scale = max(abs(weights)) / max_int_value`\n- **Quantize:** `quantized = round(weight / scale)`\n- **Dequantize:** `reconstructed = quantized * scale`\n\nFor INT8, `max_int_value` is 127 (since signed INT8 ranges from -128 to 127). So if the largest weight in a tensor is 0.85, the scale factor is 0.85 / 127 = 0.00669. A weight of 0.42 becomes round(0.42 / 0.00669) = round(62.8) = 63. To reconstruct it, you compute 63 * 0.00669 = 0.4215. The original was 0.42, so the error is 0.0015. Not bad.\n\n### Asymmetric Quantization\n\nSymmetric quantization assumes the weights are centered around zero. If they are not, for example, if a layer's weights range from -0.1 to 0.9, symmetric quantization wastes half of the integer range on values that do not exist. Asymmetric quantization adds a zero-point offset to handle this:\n\n- **Scale:** `scale = (max_val - min_val) / (max_int - min_int)`\n- **Zero-point:** `zero_point = round(min_int - min_val / scale)`\n- **Quantize:** `quantized = round(weight / scale) + zero_point`\n- **Dequantize:** `reconstructed = (quantized - zero_point) * scale`\n\nAsymmetric quantization uses the full integer range more efficiently when the weight distribution is skewed, which is common in certain layers.\n\n### Python Example\n\nLet's see this in action with a concrete weight tensor.\n\n\n```python\nimport torch\n\ndef symmetric_quantize(weights: torch.Tensor, num_bits: int = 8):\n \"\"\"Quantize weights using symmetric quantization.\"\"\"\n max_int = 2 ** (num_bits - 1) - 1 # 127 for INT8, 7 for INT4\n\n # Compute scale factor\n max_abs = weights.abs().max()\n scale = max_abs / max_int\n\n # Quantize: divide by scale and round to nearest integer\n quantized = torch.clamp(\n torch.round(weights / scale),\n -max_int - 1,\n max_int\n ).to(torch.int8)\n\n return quantized, scale\n\ndef dequantize(quantized: torch.Tensor, scale: float):\n \"\"\"Reconstruct approximate floating-point values.\"\"\"\n return quantized.float() * scale\n\n# Simulate a weight tensor from a real model layer\ntorch.manual_seed(42)\noriginal_weights = torch.randn(4, 4) * 0.5 # typical range for LLM weights\n\nprint(\"Original weights:\")\nprint(original_weights)\n\n# Quantize to INT8\nquantized_int8, scale_int8 = symmetric_quantize(original_weights, num_bits=8)\nreconstructed_int8 = dequantize(quantized_int8, scale_int8)\n\nprint(f\"\\nScale factor (INT8): {scale_int8:.6f}\")\nprint(f\"Quantized (INT8):\\n{quantized_int8}\")\nprint(f\"Reconstructed:\\n{reconstructed_int8}\")\n\n# Measure quantization error\nerror_int8 = (original_weights - reconstructed_int8).abs().mean()\nprint(f\"\\nMean absolute error (INT8): {error_int8:.6f}\")\n\n# Now quantize to INT4 for comparison\nquantized_int4, scale_int4 = symmetric_quantize(original_weights, num_bits=4)\nreconstructed_int4 = dequantize(quantized_int4, scale_int4)\n\nerror_int4 = (original_weights - reconstructed_int4).abs().mean()\nprint(f\"Mean absolute error (INT4): {error_int4:.6f}\")\nprint(f\"\\nINT4 error is {error_int4 / error_int8:.1f}x larger than INT8 error\")\n```\n\n\nThe output will show that INT8 quantization introduces very small errors (typically less than 0.002 for weights in the -0.5 to 0.5 range), while INT4 errors are roughly 10-15x larger. That sounds alarming, but remember that a neural network has billions of parameters, and these small errors tend to average out across the network. The model's overall behavior is surprisingly resilient.\n\n### Per-Channel vs Per-Tensor Quantization\n\nIn the example above, we computed one scale factor for the entire tensor. This is called per-tensor quantization. A smarter approach is per-channel quantization, where you compute a separate scale factor for each output channel (row of the weight matrix). Since different channels can have very different value ranges, per-channel quantization reduces error significantly at the cost of storing one scale factor per channel (a negligible overhead for large weight matrices).\n\nMost modern quantization tools use per-channel or even per-group quantization (where each group of 32 or 128 weights gets its own scale factor) for the best quality.\n\n---\n\n## Post-Training Quantization (PTQ)\n\nPost-Training Quantization is exactly what it sounds like: you take a fully trained FP16 model and convert its weights to a lower precision format. No retraining, no gradient computation, just a conversion step. This is by far the most popular approach because it is fast (minutes to hours, not days) and works with any model.\n\nThree PTQ formats dominate the LLM ecosystem today. Each takes a different approach to minimizing the quality loss that comes with reducing precision.\n\n### GPTQ (GPT Quantization)\n\nGPTQ, introduced by Frantar et al. in 2022, was one of the first practical methods for quantizing large language models to 4-bit or 3-bit precision. The key insight is that you can minimize quantization error by looking at weights one at a time and adjusting the remaining unquantized weights to compensate for the error introduced by quantizing the current one.\n\nGPTQ needs a small calibration dataset (typically 128-1024 samples) to measure activation patterns. It uses this data to compute a Hessian matrix that tells it which weights are most sensitive to quantization error. The algorithm then processes weights in order, quantizing each one and updating the others to minimize the total output error.\n\n**Strengths:** Good quality at INT4. Wide ecosystem support. Works well on GPUs with frameworks like vLLM, AutoGPTQ, and ExLlamaV2.\n\n**Weaknesses:** Quantization process is slower than AWQ (can take hours for large models). Slightly lower quality than AWQ on some benchmarks.\n\n### AWQ (Activation-Aware Weight Quantization)\n\nAWQ, introduced by Lin et al. in 2023, takes a different approach. Instead of trying to minimize error across all weights equally, it observes that a small fraction of weights (roughly 1%) are disproportionately important because they process activations with large magnitudes. Quantizing these critical weights introduces much more error than quantizing the rest.\n\nAWQ protects these important weights by applying per-channel scaling before quantization. It multiplies the important weight channels by a scaling factor to make them larger (and therefore less affected by rounding), then divides the corresponding activations by the same factor to keep the output unchanged. The result is that the most important weights get quantized with much less relative error.\n\n**Strengths:** Better quality than GPTQ at the same bit width. Faster quantization process. Excellent support in vLLM.\n\n**Weaknesses:** Still relatively new, so some niche frameworks may not support it yet.\n\n### GGUF (GPT-Generated Unified Format)\n\nGGUF is not a quantization algorithm. It is a file format created by the llama.cpp project that stores quantized model weights along with metadata (tokenizer, model architecture, quantization parameters) in a single file. GGUF supports many quantization levels with descriptive names like Q4_K_M, Q5_K_S, and Q8_0.\n\nThe naming convention tells you the quantization details:\n\n- **Q4, Q5, Q8:** The number of bits per weight (4-bit, 5-bit, 8-bit)\n- **K:** Uses k-quant, a method that groups weights and allows different groups to use different bit widths\n- **S, M, L:** Small, Medium, Large, indicating how many groups get higher precision (L keeps more groups at higher precision)\n\nGGUF's superpower is hardware flexibility. Models in GGUF format can run on CPUs, Apple Silicon GPUs (via Metal), NVIDIA GPUs (via CUDA), and mixed CPU/GPU setups. This makes GGUF the go-to format for local deployment with tools like Ollama and LM Studio.\n\n**Strengths:** Runs on basically any hardware. Granular quantization options. Great for local development and edge deployment.\n\n**Weaknesses:** CPU inference is slower than GPU-native formats. Not optimized for high-throughput GPU serving like AWQ/GPTQ.\n\n### PTQ Format Comparison\n\n\n| Aspect | GPTQ | AWQ | GGUF |\n|--------|------|-----|------|\n| Target hardware | GPU | GPU | CPU, GPU, Apple Silicon |\n| Quality at INT4 | Good | Better | Good (varies by quant level) |\n| Quantization speed | Slow (hours) | Fast (minutes) | Fast |\n| Inference speed (GPU) | Fast | Fast | Moderate |\n| Inference speed (CPU) | Poor | Poor | Good |\n| vLLM support | Yes | Yes (recommended) | Limited |\n| Ollama support | No | No | Yes (default) |\n| Calibration data needed | Yes (128+ samples) | Yes (small set) | No |\n| Ecosystem | Mature | Growing fast | Very mature |\n\n\nIf you are deploying on GPU servers, AWQ is the current best choice. It gives you the best quality-to-size ratio and works seamlessly with vLLM. If you are deploying locally or on edge devices, GGUF with an appropriate quantization level (Q4_K_M is a popular sweet spot) is the way to go.\n\n---\n\n## Quantization-Aware Training (QAT)\n\nPost-Training Quantization treats quantization as a separate step after training. Quantization-Aware Training flips this around: you simulate quantization during the training process itself, so the model learns to produce good outputs even when its weights are stored at lower precision.\n\n### How QAT Works\n\nDuring a QAT forward pass, weights are quantized and dequantized on the fly. The model \"sees\" the effect of quantization at every step and adjusts its weights accordingly through gradient descent. The quantize-then-dequantize operation is called a \"fake quantization\" because the actual computation still happens in floating point. The fake quantization just introduces the rounding errors that real quantization would cause, giving the model a chance to adapt to them.\n\n\n```mermaid\nflowchart LR\n W[\"FP16 Weights\"]:::primary --> FQ[\"Fake Quantize
(round to INT4)\"]:::orange\n FQ --> DQ[\"Fake Dequantize
(back to FP16)\"]:::orange\n DQ --> FWD[\"Forward Pass\"]:::teal\n FWD --> LOSS[\"Compute Loss\"]:::red\n LOSS --> GRAD[\"Backward Pass
(gradients in FP16)\"]:::green\n GRAD --> W\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nThere is a subtlety here. The rounding operation in fake quantization has zero gradient (the derivative of `round()` is zero almost everywhere). To make backpropagation work, QAT uses the Straight-Through Estimator (STE), which simply passes the gradient through the rounding operation as if it were an identity function. It is mathematically imprecise, but it works well in practice.\n\n### When QAT Is Worth It\n\nQAT requires access to training data and compute resources for retraining (or at least fine-tuning for several epochs). This makes it significantly more expensive than PTQ. Here is when the extra effort pays off:\n\n- **Extreme quantization (INT2 or INT3):** At very low bit widths, PTQ quality degrades noticeably. QAT can recover much of the lost quality because the model learns to compensate.\n- **High-stakes applications:** If even a small quality drop is unacceptable (medical, legal, financial), QAT gives you the best quality at any given bit width.\n- **Custom models you already fine-tune:** If you are already fine-tuning a model (Module 9), adding QAT to your training pipeline is incremental work, not a whole new process.\n\nFor most teams deploying standard open-source models at INT4 or INT8, PTQ (specifically AWQ) provides excellent quality without the cost of retraining. QAT is the tool you reach for when PTQ is not good enough for your specific use case.\n\n### Pseudocode Example\n\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass FakeQuantize(torch.autograd.Function):\n \"\"\"Simulates INT4 quantization during training.\"\"\"\n\n @staticmethod\n def forward(ctx, weights, num_bits=4):\n max_int = 2 ** (num_bits - 1) - 1\n scale = weights.abs().max() / max_int\n\n # Quantize then dequantize (simulate precision loss)\n quantized = torch.clamp(torch.round(weights / scale), -max_int - 1, max_int)\n dequantized = quantized * scale\n return dequantized\n\n @staticmethod\n def backward(ctx, grad_output):\n # Straight-Through Estimator: pass gradient unchanged\n return grad_output, None\n\nclass QATLinear(nn.Module):\n \"\"\"A linear layer with fake quantization for QAT.\"\"\"\n\n def __init__(self, in_features, out_features, num_bits=4):\n super().__init__()\n self.linear = nn.Linear(in_features, out_features)\n self.num_bits = num_bits\n\n def forward(self, x):\n # Apply fake quantization to weights during forward pass\n quantized_weight = FakeQuantize.apply(self.linear.weight, self.num_bits)\n return nn.functional.linear(x, quantized_weight, self.linear.bias)\n\n# During training, the model learns to work with quantized weights\nmodel = nn.Sequential(\n QATLinear(768, 3072, num_bits=4),\n nn.ReLU(),\n QATLinear(3072, 768, num_bits=4),\n)\n\n# After training, export the actual quantized weights for deployment\n```\n\n\nThis is a simplified illustration. Production QAT tools like those in PyTorch's quantization toolkit or Hugging Face's Optimum library handle the complexity of applying fake quantization to the right layers, managing different quantization schemes per layer, and exporting the final quantized model.\n\n---\n\n## Quality vs Memory vs Speed Trade-offs\n\nNot every task responds to quantization the same way. Some tasks are remarkably robust, barely losing any quality even at INT4. Others are sensitive and degrade noticeably. Understanding this is critical for deciding how aggressively you can quantize.\n\n### Why Some Tasks Are More Robust\n\nQuantization introduces small, random-ish errors into every weight. Tasks that rely on broad pattern matching (classification, extraction, summarization) are robust because the overall direction of the computation stays roughly the same even with noisy weights. Tasks that require precise numerical reasoning or creative coherence over long outputs are more sensitive because small weight errors accumulate and compound across many generation steps.\n\n### Quality Retention by Task Type\n\n\n| Task Type | FP16 (baseline) | INT8 | INT4 (AWQ) | INT4 (GPTQ) |\n|-----------|-----------------|------|------------|-------------|\n| Text classification | 100% | 99.5% | 98-99% | 97-99% |\n| Named entity extraction | 100% | 99% | 97-99% | 96-98% |\n| Summarization | 100% | 99% | 96-98% | 95-97% |\n| Factual Q&A (short answer) | 100% | 98-99% | 95-97% | 93-96% |\n| Code generation | 100% | 97-99% | 92-96% | 90-95% |\n| Math reasoning | 100% | 96-98% | 88-94% | 85-92% |\n| Creative writing (long-form) | 100% | 97-99% | 90-95% | 88-93% |\n| Multi-step reasoning | 100% | 95-98% | 85-93% | 82-90% |\n\n\nThese numbers are approximate and vary by model, dataset, and evaluation method. But the pattern is consistent: the more a task depends on precise reasoning or long-range coherence, the more it suffers from quantization.\n\n### The Memory-Speed-Quality Triangle\n\n\n```mermaid\nflowchart TD\n FP16[\"FP16
140 GB | 100% quality
Baseline speed\"]:::primary\n INT8[\"INT8
70 GB | ~98% quality
1.2-1.5x faster\"]:::teal\n INT4_AWQ[\"INT4 AWQ
35 GB | ~95% quality
1.5-2x faster\"]:::green\n INT4_GPTQ[\"INT4 GPTQ
35 GB | ~93% quality
1.5-2x faster\"]:::orange\n INT2[\"INT2
17.5 GB | ~80% quality
2-3x faster\"]:::red\n\n FP16 -->|\"2x smaller\"| INT8\n INT8 -->|\"2x smaller\"| INT4_AWQ\n INT8 -->|\"2x smaller\"| INT4_GPTQ\n INT4_AWQ -->|\"2x smaller\"| INT2\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\nAll numbers shown are for a 70B model. The speed improvement from quantization comes primarily from reduced memory bandwidth requirements. LLM inference is memory-bandwidth bound (not compute bound) during token generation, so moving less data from memory to compute units directly translates to faster generation.\n\nA practical rule of thumb: INT8 is almost always safe. You lose less than 2% quality on virtually every task, and you halve your memory requirement. INT4 (AWQ) is safe for classification, extraction, summarization, and simple Q&A. For code generation, math, and multi-step reasoning, benchmark carefully before committing to INT4.\n\n---\n\n## Practical Guide: Choosing Your Format\n\nWith multiple quantization formats and bit widths to choose from, the decision can feel overwhelming. Here is a systematic way to think about it.\n\n\n```mermaid\nflowchart TD\n START[\"What are you
deploying?\"]:::primary\n\n START --> GPU{\"Deploying on
GPU server?\"}\n START --> LOCAL{\"Deploying locally
or on CPU?\"}\n\n GPU --> QUALITY{\"Quality
critical?\"}\n LOCAL --> GGUF[\"Use GGUF
Q4_K_M or Q5_K_M\"]:::green\n\n QUALITY -->|\"Yes\"| INT8_GPU[\"Use INT8
(AWQ or FP16)\"]:::teal\n QUALITY -->|\"No\"| INT4_GPU{\"Willing to
retrain?\"}\n\n INT4_GPU -->|\"Yes\"| QAT_PATH[\"Use QAT +
INT4 export\"]:::orange\n INT4_GPU -->|\"No\"| AWQ_PATH[\"Use AWQ
INT4\"]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\n**Quick decision rules:**\n\n- **GPU server, production traffic:** AWQ INT4. Best quality at 4-bit, excellent vLLM support, fast inference.\n- **GPU server, quality-critical tasks:** INT8 (AWQ) or just use FP16 if you have the VRAM. The quality difference between INT8 and FP16 is negligible for most applications.\n- **Local development, prototyping:** GGUF Q4_K_M via Ollama. Fast setup, runs on consumer hardware, good quality.\n- **Edge deployment, mobile, embedded:** GGUF Q4_K_S or Q3_K_M. Minimize memory at the cost of some quality.\n- **Maximum quality at low bit width, have training resources:** QAT to INT4. Worth it only if PTQ quality is insufficient for your use case.\n\n---\n\n## Quantizing a Model in Practice\n\nLet's get hands-on. Here are two practical workflows: quantizing a model to GPTQ format using AutoGPTQ, and loading a pre-quantized GGUF model with llama-cpp-python.\n\n### Quantizing with AutoGPTQ\n\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\nfrom datasets import load_dataset\n\n# 1. Load the model and tokenizer\nmodel_name = \"meta-llama/Llama-3.1-8B-Instruct\"\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# 2. Configure quantization parameters\nquantize_config = BaseQuantizeConfig(\n bits=8, # INT8 quantization\n group_size=128, # per-group quantization (128 weights per group)\n desc_act=False, # disable activation reordering for speed\n)\n\n# 3. Load the model for quantization\nmodel = AutoGPTQForCausalLM.from_pretrained(\n model_name,\n quantize_config=quantize_config,\n torch_dtype=\"auto\",\n)\n\n# 4. Prepare calibration data (GPTQ needs examples to measure activation patterns)\ncalibration_data = load_dataset(\"wikitext\", \"wikitext-2-raw-v1\", split=\"train\")\ncalibration_samples = [\n tokenizer(sample[\"text\"], return_tensors=\"pt\", truncation=True, max_length=512)\n for sample in calibration_data.select(range(256))\n if len(sample[\"text\"]) > 100\n]\n\n# 5. Run quantization (this takes 30-60 minutes for an 8B model)\nmodel.quantize(calibration_samples)\n\n# 6. Save the quantized model\noutput_dir = \"./llama-3.1-8b-instruct-gptq-int8\"\nmodel.save_quantized(output_dir)\ntokenizer.save_pretrained(output_dir)\n\nprint(f\"Quantized model saved to {output_dir}\")\n```\n\n\n### Loading a GGUF Model with llama-cpp-python\n\nFor GGUF, you typically download a pre-quantized model from Hugging Face rather than quantizing yourself. The llama.cpp community maintains quantized versions of most popular models.\n\n\n```python\nfrom llama_cpp import Llama\n\n# Download a GGUF model from Hugging Face\n# Example: bartowski/Meta-Llama-3.1-8B-Instruct-GGUF\nmodel = Llama.from_pretrained(\n repo_id=\"bartowski/Meta-Llama-3.1-8B-Instruct-GGUF\",\n filename=\"Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf\",\n n_ctx=4096, # context length\n n_gpu_layers=-1, # -1 = offload all layers to GPU (if available)\n verbose=False,\n)\n\n# Generate a response\nresponse = model.create_chat_completion(\n messages=[\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n {\"role\": \"user\", \"content\": \"Explain quantization in one paragraph.\"},\n ],\n temperature=0.7,\n max_tokens=256,\n)\n\nprint(response[\"choices\"][0][\"message\"][\"content\"])\n\n# Check memory usage\nimport psutil\nprocess = psutil.Process()\nmemory_gb = process.memory_info().rss / (1024 ** 3)\nprint(f\"\\nMemory usage: {memory_gb:.2f} GB\")\n```\n\n\n### Loading an AWQ Model with vLLM\n\nIf you are using vLLM in production, loading a pre-quantized AWQ model is the simplest path.\n\n\n```python\nfrom vllm import LLM, SamplingParams\n\n# Load a pre-quantized AWQ model\n# vLLM detects the quantization format automatically\nllm = LLM(\n model=\"casperhansen/llama-3.1-8b-instruct-awq\",\n quantization=\"awq\",\n gpu_memory_utilization=0.90,\n max_model_len=4096,\n)\n\n# Generate\nsampling_params = SamplingParams(temperature=0.7, max_tokens=256)\noutputs = llm.generate(\n [\"Explain the difference between GPTQ and AWQ quantization.\"],\n sampling_params,\n)\n\nprint(outputs[0].outputs[0].text)\n```\n\n\n---\n\n## Exercise\n\n**Quantize and Benchmark Llama 3.1 8B**\n\nTake Llama 3.1 8B Instruct and create three versions for comparison:\n\n1. **FP16 baseline:** Load the original model (requires ~16GB VRAM)\n2. **INT8 GPTQ:** Quantize using AutoGPTQ with 8-bit precision and 128-group size, using 256 calibration samples from WikiText\n3. **INT4 GGUF Q4_K_M:** Download the pre-quantized GGUF version\n\nFor each version, measure:\n\n- **Memory usage:** Peak GPU VRAM (or system RAM for GGUF on CPU)\n- **Inference speed:** Tokens per second on a batch of 50 prompts (mix of short and long outputs)\n- **Quality:** Run all three versions through an evaluation suite of 100 questions covering classification, summarization, factual Q&A, and code generation. Use an LLM-as-judge approach (from Module 10) to rate output quality on a 1-5 scale\n\nBuild a comparison table and answer these questions:\n\n- At what quantization level does quality drop become noticeable for each task type?\n- What is the memory savings vs quality trade-off for your specific workload?\n- Which format would you choose for production deployment and why?\n\n---\n\n## Summary\n\nQuantization is one of the most practical tools in the AI engineer's toolkit. It lets you run larger models on cheaper hardware, serve more concurrent users on the same GPU, and deploy models to environments (laptops, edge devices) that would otherwise be impossible.\n\nHere are the key takeaways:\n\n- **Number precision directly determines memory cost.** FP32 uses 4 bytes per parameter, FP16 uses 2 bytes, INT8 uses 1 byte, INT4 uses 0.5 bytes. A 70B model goes from 280GB (FP32) to 140GB (FP16) to 70GB (INT8) to 35GB (INT4).\n- **INT8 is almost always safe.** Quality loss is typically under 2% across all task types. If you are deploying a self-hosted model, INT8 should be your baseline, not FP16.\n- **INT4 is safe for many tasks, sensitive for others.** Classification, extraction, and summarization retain 95%+ quality. Math reasoning, code generation, and multi-step reasoning can drop to 85-93%. Always benchmark on your specific tasks.\n- **AWQ is the best PTQ format for GPU inference.** It protects important weights from quantization damage and works seamlessly with vLLM.\n- **GGUF is the best format for local and CPU deployment.** Use Q4_K_M as a default, upgrade to Q5_K_M or Q8_0 if quality matters more than memory.\n- **QAT is only worth it at extreme bit widths or for high-stakes applications.** For most INT4 and INT8 deployments, PTQ is sufficient.\n- **Benchmark before committing.** The quality retention numbers in this chapter are typical ranges, not guarantees. Your specific model, dataset, and task will determine the actual trade-off.\n\nQuantization makes models smaller by reducing the precision of their weights. But what if you need a fundamentally smaller model that is just as smart? A model with fewer parameters, not just smaller numbers? That is where knowledge distillation comes in, and it is the topic of our next chapter.\n\n---\n\n## References\n\n- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (Lin et al., 2023)](undefined)\n- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (Frantar et al., 2022)](undefined)\n- [AutoGPTQ Documentation](undefined)\n- [llama.cpp and GGUF Format](undefined)\n- [Hugging Face Quantization Guide](undefined)\n- [A Survey of Quantization Methods for Efficient Neural Network Inference (Gholami et al., 2021)](undefined)","pageType":"ai-engineering"}

11.8 Quantization

Ashish Pratap Singh

Get Premium