Knowledge Distillation

An ensemble of five models might boost accuracy by a couple of points, but it also multiplies your serving cost. You now need five times the compute, five times the memory, and a much more complex system to maintain.

Knowledge distillation offers a simpler path.

Instead of serving the entire ensemble, you train a smaller “student” model to mimic the predictions of the larger “teacher” model. Once trained, you deploy only the student. It retains most of the teacher’s accuracy while being much cheaper and easier to run in production.