Model optimization is the work of reducing inference latency, memory, and cost while preserving enough quality for the product. It is an engineering trade-off between quality, hardware, traffic volume, and operating cost, and the best choice shifts as any of those change.

The discipline is to measure the bottleneck first, then apply the least invasive optimization that hits the latency or cost target. Distilling a model when the real problem is an unbatched serving loop wastes weeks and usually makes the system harder to reason about.