{"title":"11.9 Knowledge Distillation","description":"","content":"# 11.9 Knowledge Distillation\n\n## The Problem\n\nThroughout this module, you have been cutting costs by optimizing prompts (11.2), routing to cheaper models (11.3), reducing latency (11.4), and self-hosting (11.5). You even learned how to quantize models in the previous chapter to squeeze them onto cheaper hardware. But quantization has a fundamental limitation: it compresses the same model into fewer bits. A 7B parameter model quantized to 4-bit is still a 7B parameter model. It runs faster and uses less memory, but it is not any smarter than it was at full precision.\n\nWhat if you need GPT-4-level quality but at GPT-4o-mini costs? Not just a compressed version of a big model, but a genuinely smaller model that punches above its weight class. A model with fewer layers, fewer parameters, and faster inference that somehow retains most of the large model's capability.\n\nThis is the core idea behind knowledge distillation. You take a large, expensive \"teacher\" model, and you train a smaller, cheaper \"student\" model to mimic its behavior. The student will never be quite as good as the teacher, but it can get surprisingly close, often retaining 95-97% of the teacher's quality at a fraction of the cost and latency.\n\n**What you will learn:**\n\n- Why the teacher's probability distribution contains richer information than just the correct answer\n- How the distillation loss function combines ground truth and teacher knowledge\n- Classic distillation results (DistilBERT, TinyLlama) and what they achieved\n- Modern API-based distillation: using GPT-4o outputs to fine-tune GPT-4o-mini\n- When to use distillation vs. quantization\n- Task-specific vs. general distillation trade-offs\n- Legal and licensing considerations for production distillation\n\n---\n\n## How Distillation Works\n\nThe basic setup is straightforward. You have two models:\n\n- **Teacher model:** Large, expensive, high quality. This might be GPT-4o, Claude Sonnet, or Llama 3.1 70B. It produces excellent outputs but costs too much to run at scale.\n- **Student model:** Small, cheap, fast. This might be GPT-4o-mini, Llama 3.1 8B, or even a custom 1B parameter model. It runs cheaply but does not produce outputs as good as the teacher.\n\nThe goal is to train the student to behave like the teacher. You feed the same inputs to both models, then adjust the student's weights so its outputs match the teacher's as closely as possible.\n\nBut here is the key insight that makes distillation more powerful than simply training on the correct answers. When a teacher model processes the prompt \"What is the capital of France?\", it does not just output \"Paris.\" It produces a probability distribution over its entire vocabulary. That distribution might look something like this:\n\n\n| Token | Probability |\n|-------|------------|\n| Paris | 72% |\n| Lyon | 15% |\n| Marseille | 8% |\n| Toulouse | 3% |\n| Other | 2% |\n\n\nThe correct answer is \"Paris,\" and if you were just training on labels, you would give the student a one-hot vector: 100% for Paris, 0% for everything else. But look at how much information you are throwing away. The teacher is telling you that Lyon and Marseille are related to the question, that they are French cities, that Lyon is more likely than Marseille. This \"dark knowledge\" (a term coined by Geoffrey Hinton, who introduced the technique) encodes relationships between concepts that the student can learn from.\n\nBy training the student to match the full probability distribution, not just the top answer, you transfer a much richer understanding of the task.\n\n\n```mermaid\nflowchart TD\n INPUT[Input Data]:::primary --> TEACHER[Teacher Model
Large, Expensive]:::orange\n INPUT --> STUDENT[Student Model
Small, Fast]:::primary\n\n TEACHER --> SOFT[Soft Probability
Distribution]:::teal\n TEACHER --> HARD_T[Hard Labels
Correct Answer]:::green\n\n STUDENT --> PRED[Student
Predictions]:::primary\n\n SOFT --> LOSS[Combined
Distillation Loss]:::purple\n HARD_T --> LOSS\n PRED --> LOSS\n\n LOSS --> UPDATE[Update Student
Weights]:::orange\n UPDATE --> STUDENT\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\nThe student learns from two signals simultaneously: the ground truth labels (what the correct answer actually is) and the teacher's soft probabilities (what the teacher thinks about all possible answers). This dual signal is what makes distillation so effective.\n\n---\n\n## The Distillation Loss Function\n\nThe loss function for distillation has two components, and understanding how they work together is essential.\n\n**Component 1: Standard Cross-Entropy Loss.** This is the normal training loss. Compare the student's prediction against the ground truth label. This keeps the student grounded in reality, the teacher might have biases or errors, and the ground truth loss prevents the student from inheriting them blindly.\n\n**Component 2: KL Divergence Loss.** This measures how different the student's probability distribution is from the teacher's. KL divergence (Kullback-Leibler divergence) is zero when the two distributions are identical and grows as they diverge. This is the term that actually transfers the teacher's knowledge.\n\nThere is one more ingredient: **temperature**. Neural networks typically produce very \"sharp\" probability distributions where one token gets 99% of the probability and everything else gets nearly zero. That does not leave much for the student to learn from. Temperature scaling softens the distribution by dividing the logits (raw model outputs before softmax) by a temperature value T before applying softmax.\n\n- **T = 1:** Normal, sharp distribution. \"Paris\" gets 99%.\n- **T = 5:** Softer distribution. \"Paris\" gets 60%, \"Lyon\" gets 20%, \"Marseille\" gets 12%.\n- **T = 20:** Very soft. Probabilities are nearly uniform, revealing subtle relationships.\n\nHigher temperatures expose more of the internal structure the teacher has learned. In practice, temperatures between 2 and 10 work well for most tasks.\n\nThe combined loss looks like this:\n\n**L = alpha * CrossEntropy(student, ground_truth) + (1 - alpha) * T^2 * KL_Divergence(student_soft, teacher_soft)**\n\nThe T^2 scaling factor compensates for the fact that softened distributions produce smaller gradients. The alpha parameter controls the balance between learning from ground truth vs. learning from the teacher. A typical starting point is alpha = 0.5.\n\nHere is a PyTorch implementation:\n\n\n```python\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\n\nclass DistillationLoss(nn.Module):\n def __init__(self, temperature=4.0, alpha=0.5):\n super().__init__()\n self.temperature = temperature\n self.alpha = alpha\n self.ce_loss = nn.CrossEntropyLoss()\n\n def forward(self, student_logits, teacher_logits, labels):\n # Component 1: Standard cross-entropy with ground truth\n hard_loss = self.ce_loss(student_logits, labels)\n\n # Component 2: KL divergence between soft distributions\n # Soften both distributions with temperature\n student_soft = F.log_softmax(student_logits / self.temperature, dim=-1)\n teacher_soft = F.softmax(teacher_logits / self.temperature, dim=-1)\n\n # KL divergence, scaled by T^2 to balance gradients\n soft_loss = F.kl_div(\n student_soft,\n teacher_soft,\n reduction=\"batchmean\"\n ) * (self.temperature ** 2)\n\n # Combine both components\n total_loss = self.alpha * hard_loss + (1 - self.alpha) * soft_loss\n return total_loss\n\n# Usage example\ntemperature = 4.0\nalpha = 0.5\ndistill_loss = DistillationLoss(temperature=temperature, alpha=alpha)\n\n# Simulated logits (batch_size=2, vocab_size=5)\nstudent_logits = torch.randn(2, 5)\nteacher_logits = torch.randn(2, 5)\nlabels = torch.tensor([0, 3])\n\nloss = distill_loss(student_logits, teacher_logits, labels)\nprint(f\"Distillation loss: {loss.item():.4f}\")\n# Distillation loss: 2.1437\n```\n\n\nThe `log_softmax` and `softmax` pairing is important. PyTorch's `kl_div` expects log-probabilities for the first argument and regular probabilities for the second. Getting this wrong is a common bug that produces training that looks like it is working but converges to a bad result.\n\n**Exercise:** Modify the `DistillationLoss` class to experiment with different temperatures (1, 4, 10, 20). For a fixed pair of teacher and student logits, print the soft probability distributions at each temperature. Observe how higher temperatures make the distributions more uniform and reveal more structure.\n\n---\n\n## Classic Distillation Examples\n\nDistillation is not just a theoretical idea. Some of the most widely deployed models in production are distilled versions of larger ones.\n\n**DistilBERT** (Hugging Face, 2019) is probably the most famous example. The team at Hugging Face distilled BERT-base (110M parameters) into DistilBERT (66M parameters) using the exact technique described above. The results were striking:\n\n\n| Metric | BERT-base | DistilBERT | Change |\n|--------|-----------|------------|--------|\n| Parameters | 110M | 66M | 40% smaller |\n| Inference speed | 1x | 1.6x | 60% faster |\n| GLUE benchmark | 100% | 97% | 3% loss |\n\n\nA 40% reduction in size and 60% speedup for only a 3% quality drop. For most production applications, that trade-off is extremely attractive. DistilBERT became one of the most downloaded models on Hugging Face, with teams using it in production for classification, NER, and embedding generation where latency and cost matter more than squeezing out the last few percentage points of accuracy.\n\n**TinyLlama** (Zhang et al., 2024) took a different approach. Instead of distilling from a specific teacher checkpoint, the team trained a 1.1B parameter model on the Llama 2 architecture using a massive 3 trillion token dataset. While not traditional distillation (it used standard pretraining rather than teacher soft labels), it borrowed the architectural insights from larger Llama models. The result was a 1.1B model that performed competitively with much larger models on many benchmarks, small enough to run on a phone.\n\n**Gemma 2 2B** (Google, 2024) used distillation from larger Gemma models and achieved remarkable results for its size. Google explicitly noted that distillation from their larger models was key to the 2B variant's performance, which outperformed models 2-3x its size on several benchmarks.\n\nThe pattern across all these examples is consistent: distillation reliably transfers 90-97% of a teacher's capability into a model that is 2-10x smaller and faster.\n\n---\n\n## Modern API-Based Distillation\n\nThe classic distillation approach requires access to the teacher model's logits, the raw probability distributions over the vocabulary. That works great when you are distilling open-source models where you have full weight access. But what about proprietary models like GPT-4o or Claude Sonnet? You cannot extract their logits through an API.\n\nThis is where modern API-based distillation comes in, and it is how most practitioners actually do distillation today. The idea is simple:\n\n1. Use the teacher model's API to generate high-quality training data\n2. Fine-tune a smaller student model on that data\n\nYou lose the soft probability distributions (you only get the teacher's top-1 output, not its full distribution). But in practice, the teacher's outputs are still much richer training signal than typical human-written labels because they are consistent, detailed, and reflect the teacher's reasoning patterns.\n\n\n```mermaid\nflowchart LR\n PROMPTS[Task Prompts
1,000+ Examples]:::primary --> TEACHER_API[Teacher API
GPT-4o]:::orange\n TEACHER_API --> DATASET[Training Dataset
Input-Output Pairs]:::teal\n DATASET --> FINETUNE[Fine-Tune
Student Model]:::green\n FINETUNE --> STUDENT[Student Model
GPT-4o-mini]:::primary\n\n EVAL[Evaluation Set
100 Examples]:::purple --> COMPARE{Compare
Quality}:::orange\n STUDENT --> COMPARE\n TEACHER_API --> COMPARE\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef purple fill:#9775fa,stroke:#000,color:#000\n```\n\n\nHere is a complete workflow that uses GPT-4o as the teacher to generate training data, then fine-tunes GPT-4o-mini as the student:\n\n\n```python\nimport json\nimport random\nfrom openai import OpenAI\n\nclient = OpenAI()\n\n# Step 1: Define your task prompts\n# These should represent the actual inputs your system will handle\ntask_prompts = [\n \"Review this Python function for bugs and style issues:\\n\\ndef calculate_avg(nums):\\n total = 0\\n for n in nums:\\n total += n\\n return total / len(nums)\",\n \"Review this Python function for bugs and style issues:\\n\\ndef fetch_user(user_id):\\n resp = requests.get(f'http://api.example.com/users/{user_id}')\\n return resp.json()\",\n # ... imagine 1,000 such prompts for your specific task\n]\n\n# Step 2: Generate teacher outputs\ndef generate_teacher_outputs(prompts, model=\"gpt-4o\"):\n \"\"\"Use the teacher model to generate high-quality responses.\"\"\"\n training_examples = []\n\n for i, prompt in enumerate(prompts):\n response = client.chat.completions.create(\n model=model,\n messages=[\n {\n \"role\": \"system\",\n \"content\": \"You are an expert code reviewer. Provide detailed, \"\n \"actionable feedback on code quality, bugs, and style.\"\n },\n {\"role\": \"user\", \"content\": prompt}\n ],\n temperature=0.7, # slight variation for diverse outputs\n max_tokens=1024\n )\n\n training_examples.append({\n \"messages\": [\n {\n \"role\": \"system\",\n \"content\": \"You are an expert code reviewer. Provide detailed, \"\n \"actionable feedback on code quality, bugs, and style.\"\n },\n {\"role\": \"user\", \"content\": prompt},\n {\"role\": \"assistant\", \"content\": response.choices[0].message.content}\n ]\n })\n\n if (i + 1) % 100 == 0:\n print(f\"Generated {i + 1}/{len(prompts)} examples\")\n\n return training_examples\n\n# Step 3: Save as JSONL for fine-tuning\ndef save_training_data(examples, filepath=\"training_data.jsonl\"):\n \"\"\"Save in OpenAI fine-tuning format.\"\"\"\n # Shuffle and split into train/validation\n random.shuffle(examples)\n split_idx = int(len(examples) * 0.9)\n train = examples[:split_idx]\n val = examples[split_idx:]\n\n for name, data in [(\"train\", train), (\"val\", val)]:\n path = filepath.replace(\".jsonl\", f\"_{name}.jsonl\")\n with open(path, \"w\") as f:\n for example in data:\n f.write(json.dumps(example) + \"\\n\")\n print(f\"Saved {len(data)} examples to {path}\")\n\n return filepath.replace(\".jsonl\", \"_train.jsonl\")\n\n# Step 4: Fine-tune the student model\ndef fine_tune_student(training_file_path, model=\"gpt-4o-mini-2024-07-18\"):\n \"\"\"Upload data and start fine-tuning.\"\"\"\n # Upload the training file\n with open(training_file_path, \"rb\") as f:\n file_obj = client.files.create(file=f, purpose=\"fine-tune\")\n\n # Create the fine-tuning job\n job = client.fine_tuning.jobs.create(\n training_file=file_obj.id,\n model=model,\n hyperparameters={\n \"n_epochs\": 3,\n \"learning_rate_multiplier\": 1.8,\n }\n )\n\n print(f\"Fine-tuning job created: {job.id}\")\n print(f\"Status: {job.status}\")\n return job\n\n# Run the pipeline\nexamples = generate_teacher_outputs(task_prompts)\ntrain_path = save_training_data(examples)\njob = fine_tune_student(train_path)\n```\n\n\nThe cost math is compelling. Generating 1,000 training examples with GPT-4o costs roughly $5-15 depending on output length. Fine-tuning GPT-4o-mini costs a few dollars more. After that, you have a student model that runs at GPT-4o-mini pricing ($0.15/$0.60 per million tokens) but produces outputs that are significantly closer to GPT-4o quality ($2.50/$10.00 per million tokens) for your specific task. The one-time investment of $10-20 saves you thousands per month at scale.\n\n**Exercise:** Pick a specific task your application handles (summarization, classification, code review, data extraction, etc.). Write 50 diverse input prompts for that task. Use GPT-4o to generate responses for all 50. Manually review 10 of the teacher's outputs for quality. Are they good enough to train on? If not, refine your system prompt and regenerate.\n\n---\n\n## When Distillation Beats Quantization\n\nBoth distillation and quantization make models cheaper to run. But they solve fundamentally different problems, and picking the wrong one wastes time and money.\n\n**Quantization** takes an existing model and represents its weights with fewer bits. A 7B parameter model in FP16 uses 14GB of memory. Quantize it to 4-bit and it fits in 3.5GB. The architecture is identical, the number of parameters is identical, but each parameter takes less space. The model runs faster (less memory to move around) and fits on cheaper hardware. The downside: you lose some precision, and the quality degrades slightly. But you cannot make the model fundamentally smarter or smaller. A quantized 7B model is still doing 7B parameters worth of computation per token.\n\n**Distillation** creates a genuinely different (smaller) model. A distilled 1B parameter model has fewer layers, fewer attention heads, and a smaller hidden dimension than its 7B teacher. It does less computation per token, which means lower latency and lower cost. The trade-off is a larger quality gap than quantization, but the inference savings are also much larger.\n\n\n| Factor | Quantization | Distillation |\n|--------|-------------|-------------|\n| Architecture | Same model, fewer bits | Different (smaller) model |\n| Parameter count | Unchanged | Reduced (often 2-10x) |\n| Quality retention | 95-99% typical | 90-97% typical |\n| Inference speedup | 1.5-3x | 2-10x |\n| Memory reduction | 2-4x | 2-10x |\n| Requires training | No (post-processing) | Yes (fine-tuning) |\n| Setup effort | Minutes | Hours to days |\n| Can combine both | Yes | Yes |\n\n\nThe best approach often combines both. Distill a 70B model into a 7B model, then quantize the 7B to 4-bit. You get the architectural efficiency of the smaller model plus the memory savings of quantization.\n\n**When to choose quantization:** You are happy with your model's capability and just need it to run on cheaper hardware or with lower latency. You do not have time or data for training. You need a quick win.\n\n**When to choose distillation:** You need a fundamentally smaller model (fewer parameters, not just fewer bits). You have a specific task where a small model can match a large one. You are optimizing for the lowest possible per-request cost at high volume.\n\n**When to choose both:** You are deploying at massive scale and every millisecond and every megabyte matters. Distill first to get the smallest model that meets your quality bar, then quantize to squeeze it further.\n\n---\n\n## Task-Specific vs. General Distillation\n\nThere are two flavors of distillation, and the distinction matters a lot for practitioners.\n\n**General distillation** tries to create a student that replicates the teacher across all tasks. This is what DistilBERT did: it learned to mimic BERT on every kind of input, from sentiment analysis to question answering to named entity recognition. General distillation requires enormous training datasets (millions of diverse examples) and significant compute. The result is a versatile smaller model, but the quality gap with the teacher is spread across all capabilities.\n\n**Task-specific distillation** focuses on a single domain or task. Instead of teaching the student everything, you teach it one thing really well. A code review distillation uses only code review examples. A customer support distillation uses only support conversations. This approach needs far less data (often 500-2,000 examples), trains faster, and frequently produces better results on the target task than general distillation does.\n\nMost production use cases are task-specific. Here is why. When you deploy an LLM in production, it is rarely doing \"everything.\" It is doing one thing: answering customer questions, reviewing code, extracting data from documents, or generating summaries. You do not need a student that can write poetry if your application only does data extraction. By focusing the training data on your specific task, the student can allocate all of its limited capacity to that one thing.\n\nThe practical guideline is:\n\n- **< 1,000 examples:** Task-specific distillation is your only realistic option. Focus on one task.\n- **1,000 - 10,000 examples:** You can cover a handful of related tasks (e.g., code review + bug detection + code explanation).\n- **> 100,000 examples:** General distillation becomes feasible, but still expect a larger quality gap than task-specific.\n\nFor the API-based distillation workflow we covered earlier, task-specific is almost always the right choice. Generating 1,000 focused examples is cheap and fast. Generating 100,000 diverse examples for general distillation costs hundreds of dollars in API calls and takes days.\n\n---\n\n## Legal and Licensing Considerations\n\nThis section is not the most exciting, but it can save you from serious legal problems. When you use a model's API outputs as training data for another model, you are subject to that provider's terms of service. Different providers have very different policies, and they change frequently.\n\n**OpenAI** (as of early 2025) allows using their API outputs to train models, including competing models. Their terms of service explicitly state that output from the API belongs to the customer. This makes GPT-4o a popular teacher model for distillation. However, there is an important nuance: you cannot use outputs from ChatGPT (the consumer product) for training, only from the API.\n\n**Anthropic** has similar terms for API usage. Outputs generated through the API can be used for training, subject to their acceptable use policy. Again, the key distinction is API usage vs. consumer product usage.\n\n**Google** allows using Gemini API outputs for training in most cases, but their terms have more restrictions around certain use cases. Check their current terms before relying on Gemini outputs for production training data.\n\n**Open-source models** (Llama, Mistral, Qwen) generally have the fewest restrictions. Their licenses typically allow using model outputs for any purpose, including training other models. If licensing risk is a concern, using an open-source teacher model eliminates the issue entirely.\n\nA few practical guidelines:\n\n- Always use the **API**, not the consumer chat product, when generating training data\n- **Document your process.** Keep records of which model generated which data, when, and under what terms\n- **Re-check terms periodically.** Provider policies evolve. What is allowed today might change next quarter\n- **When in doubt, use open-source teachers.** Llama 3.1 70B is an excellent teacher model with permissive licensing\n\nThis is not legal advice. If your organization is building production systems that depend on distilled models, have your legal team review the specific terms of service for whatever teacher model you plan to use.\n\n---\n\n## Exercise\n\nBuild an end-to-end API-based distillation pipeline:\n\n1. **Choose a specific task** for your student model (e.g., code review, email summarization, data extraction from receipts, SQL query generation).\n\n1. **Write 1,000 diverse input prompts** for that task. Aim for variety: different input lengths, edge cases, different difficulty levels.\n\n1. **Generate teacher outputs** using GPT-4o (or another strong model). Use a well-crafted system prompt that produces the output format you want.\n\n1. **Quality check** a random sample of 50 teacher outputs. Score them on a 1-5 scale. If the average is below 4, refine your system prompt and regenerate.\n\n1. **Fine-tune GPT-4o-mini** on the teacher's outputs using the OpenAI fine-tuning API.\n\n1. **Evaluate the student** on a held-out test set of 100 examples. Compare:\n\n- **Quality:** Use GPT-4o as a judge to rate both the teacher's and student's outputs on a 1-5 scale\n- **Latency:** Measure time-to-first-token and total response time for both models\n- **Cost:** Calculate per-request cost for teacher vs. student\n\n1. **Document the quality-cost trade-off.** What percentage of the teacher's quality did the student retain? What is the cost savings? Would you deploy the student in production?\n\n---\n\n## Summary\n\nHere are the key takeaways from this chapter:\n\n- **Knowledge distillation transfers capability from a large teacher model to a smaller student model.** The student learns from the teacher's full probability distribution, not just the correct answers, capturing richer information about relationships between concepts.\n- **The distillation loss combines two signals:** cross-entropy against ground truth labels and KL divergence against the teacher's soft probabilities. Temperature scaling controls how much structure the soft distributions reveal, with values between 2 and 10 working well for most tasks.\n- **Classic results prove the technique works at scale.** DistilBERT retained 97% of BERT's quality at 40% smaller and 60% faster. These are not theoretical numbers, they are production-validated across thousands of deployments.\n- **API-based distillation is the modern practitioner's approach.** Generate training data through the teacher's API, then fine-tune a smaller model on those outputs. You lose the soft probability distributions but still get a powerful training signal. The one-time cost is $10-20 for a task-specific dataset that saves thousands monthly.\n- **Distillation and quantization solve different problems.** Quantization reduces bit-width (same architecture, fewer bits). Distillation reduces architecture size (fewer parameters, full precision). Combine both for maximum efficiency.\n- **Task-specific distillation outperforms general distillation** in almost all production settings. With 500-2,000 focused examples, a student can match 95%+ of the teacher's quality on a specific task.\n- **Check licensing before you distill.** OpenAI and Anthropic currently allow API outputs for training, but terms change. Open-source teacher models eliminate licensing risk entirely.\n\nDistillation creates smaller, specialized models by transferring knowledge from larger ones. But what if the model itself could be massive in total parameters, yet only activate a small fraction of them for each query? Instead of making one small model work hard on everything, you could have many small expert models, each handling the queries they are best at. That is the idea behind Mixture of Experts, which we will cover next.\n\n---\n\n## References\n\n- [Distilling the Knowledge in a Neural Network (Hinton et al., 2015)](undefined)\n- [DistilBERT, a distilled version of BERT (Sanh et al., 2019)](undefined)\n- [TinyLlama: An Open-Source Small Language Model (Zhang et al., 2024)](undefined)\n- [OpenAI Fine-tuning Documentation](undefined)\n- [A Survey of Knowledge Distillation in Large Language Models (Xu et al., 2024)](undefined)","pageType":"ai-engineering"}

11.9 Knowledge Distillation

Ashish Pratap Singh

Get Premium