Here is the uncomfortable truth: most of your queries do not need a big model like GPT-4o. When a user asks "What is the capital of France?" or "Convert 5 miles to kilometers," you are paying premium prices for a task that a model 10x cheaper could handle just as well.

In a typical production application, somewhere between 60% and 80% of queries are straightforward enough for a smaller, cheaper model. Only the remaining 20-40% actually benefit from the reasoning power of a frontier model.

The solution is not to downgrade everything to a cheap model. That would tank quality on complex queries. The solution is to build a routing layer that sends each query to the right model for the job.

Simple questions go to a fast, cheap model. Complex reasoning goes to a powerful, expensive one. And queries you have already answered get served from a cache without hitting any model at all. This is how companies like Notion, Cursor, and Intercom keep their AI features affordable at scale.