Knowledge Distillation

An ensemble of five models may improve accuracy, but it also multiplies serving cost. You now need more compute, more memory, more deployment plumbing, and more monitoring.

Knowledge distillation is one way to keep much of that quality while serving a smaller model.

Instead of serving the entire teacher system, you train a smaller "student" model to mimic it. Once trained, you deploy the student. A good student usually recovers a large fraction of the teacher's advantage at a much lower serving cost.