Every GenAI system has to decide which model, or models, to use.

There are plenty of options: hosted frontier models, open-weight models you can run yourself, smaller models optimized for speed, reasoning models, multimodal models, and models with context windows large enough to process an entire book.

Choosing the right model means balancing several requirements: answer quality, response time, cost per request, data privacy, and how much infrastructure your team can manage. A system can also use multiple models, choosing a different one for each type of task.

In this chapter, we'll look at how to make these decisions, from choosing between hosted and self-hosted models to deciding when long context makes sense and when retrieval is a better option.

Public benchmarks measure how well models perform on general tasks, but your system has its own requirements and constraints. Before comparing models, identify what matters most for your application:

Some requirements are non-negotiable. For example, if sensitive data must remain within your company's network, external hosted APIs may not be an option, regardless of how well they perform. Other requirements involve trade-offs between quality, speed, and cost.

A practical approach is to filter out models that don't meet your hard requirements, evaluate the remaining candidates on your actual tasks, and then compare their latency and cost.

This process helps you narrow down your options without relying entirely on benchmark scores. Hard constraints eliminate models you can't use, evaluations show which models meet your quality requirements, and latency and cost help you choose between them.

One of the first decisions you'll need to make is whether to use a hosted model through an API or deploy and run the model yourself.

With hosted APIs, you access models from providers like OpenAI, Anthropic, and Google through an API. Cloud platforms like AWS Bedrock, Google Vertex AI, and Azure also offer managed model access. The provider handles GPUs, scaling, and infrastructure, while you typically pay based on token usage.

Open-weight models, such as Llama, Qwen, Mistral, and DeepSeek, make their weights available for download. You can run them on your own infrastructure using serving engines like vLLM or SGLang. You can also use a third-party inference provider to access these models through an API without managing GPUs yourself.

Here's how the two approaches compare:

The Cost Depends on GPU Utilization

Self-hosting can be cheaper at scale, but only if you keep your GPUs busy.

When you self-host, you pay for GPU capacity even when it's sitting idle. Suppose a GPU server costs $20 per hour and can process 20 million tokens per hour at full capacity.

The cost per million tokens depends on how much of that capacity you actually use:

In practice, traffic fluctuates. Your application might receive thousands of concurrent requests during peak hours but very few overnight. If you provision enough GPUs for peak traffic, average utilization might be only 30 to 40 percent.

You also need extra capacity for failures and maintenance, plus engineers to manage the infrastructure.

Now suppose a hosted API costs $1.50 per million tokens for an equivalent workload. At 30 percent GPU utilization, the hosted API is cheaper. At full utilization, self-hosting wins on infrastructure cost.

Of course, real costs also depend on the model, hardware, batching efficiency, and input-to-output token ratio. In general, self-hosting becomes more economical when traffic is high and predictable enough to keep GPUs busy.

That's why it can work well for large-scale batch processing or applications with steady demand, but is often harder to justify for a new feature with uncertain usage.

When to Choose Each Approach

Starting with a hosted API is usually the simpler choice. You can launch quickly, evaluate different models, and understand your actual traffic and costs before investing in GPU infrastructure.

You can also combine both approaches. For example, you might self-host a fine-tuned 8B model for high-volume classification tasks while using a hosted frontier model for requests that need more advanced reasoning.

For each task, choose the most cost-effective option that meets its quality, latency, and operational requirements.

That leads to another important decision: how large does the model actually need to be?

Not every task needs a large model. A single user request might involve several model calls, each with different requirements.

For example, intent classification, search query rewriting, and basic content checks are relatively simple tasks that smaller models can often handle well. Generating a detailed answer from multiple documents, however, may benefit from a larger model.

Here's how you could combine them:

Here, two of the three model calls use smaller, cheaper models. Only answer generation uses the larger model, where its additional capabilities are more valuable.

You can go a step further with model routing. Instead of using the same model for every request, a classifier or a set of rules can route simple questions to a smaller model and harder ones to a more capable model.

The best way to choose is through evaluation. Start with a capable model to establish a quality baseline. Then test smaller models on the same evaluation set and compare quality, latency, and cost.

If a smaller model costs one-fifth as much while delivering nearly the same quality, it's likely the better choice for that task. Just remember that adding more model calls can also increase end-to-end latency, so evaluate the entire request flow.

Use the smallest model that reliably meets your quality requirements.

Model size isn't the only way to improve performance, though. Reasoning models introduce another trade-off: spending more computation during inference to solve difficult problems, rather than relying on model size alone.

A standard model typically generates its answer without an extended reasoning phase. A reasoning model, also called a thinking model, uses additional computation to work through a problem before producing its final response. This is known as test-time compute, where spending more computation during inference can improve the quality of the answer.

Reasoning models are especially useful for complex tasks involving multiple steps, such as debugging code, solving math problems, planning agent actions, or analyzing legal documents. For simpler tasks, like extracting a date from an email or answering a common FAQ, the extra reasoning often isn't worth the cost.

The Cost of Additional Reasoning

Reasoning models generate additional tokens, which are generally billed as output tokens. Even when a provider doesn't expose the full reasoning process, those tokens may still count toward your bill.

Suppose a model charges $15 per million output tokens. Here's how reasoning effort could affect the cost of a request:

These are illustrative estimates, assuming 100 reasoning tokens per second and excluding input token costs. Actual latency depends on the model and provider.

The extra latency can be just as important as the cost. Users may have to wait several seconds before the model starts producing its final answer.

In a chat application, you can show a "thinking" indicator while the model works. But for latency-sensitive applications like real-time voice assistants, making users wait ten or twenty seconds before responding is usually unacceptable.

How to Control Reasoning Costs

Many reasoning models let you configure a reasoning effort level or token budget, giving you some control over how much computation they use.

There are three practical ways to manage this:

Use reasoning models when additional computation meaningfully improves quality and the application can tolerate the extra cost and latency.

So far, we've compared models based on size, capabilities, latency, and cost. Another important factor is the model's architecture, particularly when you're hosting it yourself.

In a dense model, nearly all parameters are used to process every token. For example, a 70B dense model uses roughly 70 billion parameters during each forward pass, which requires substantial computation.

A Mixture-of-Experts (MoE) model works differently. Instead of a single feed-forward network in certain transformer layers, it has multiple networks called experts. A router selects which experts process each token, so only a subset of the model's parameters is used.

Here's a simplified example with four experts, where the router selects two for each token:

This lets MoE models have many more total parameters than they use for any single token. The parameters used are called active parameters.

For example, Mixtral 8x7B has around 47 billion total parameters but uses only about 13 billion per token. DeepSeek-V3 is much larger, with 671 billion total parameters but only about 37 billion active per token.

What This Means for Model Serving

MoE models reduce computation by activating fewer parameters, but that comes with a trade-off: they still need memory for all their weights.

The main challenge with MoE models is GPU memory. DeepSeek-V3, for example, activates only about 37 billion parameters per token, but efficient serving still requires keeping its 671 billion parameters available in memory. Different tokens may need different experts, so the model typically has to be distributed across multiple GPUs, sometimes across several machines.

This also introduces communication overhead as tokens are routed between GPUs. As a result, fewer active parameters don't automatically mean lower latency.

With hosted APIs, these infrastructure details are mostly hidden. What matters to you is the model's quality, latency, and cost.

Beyond the underlying architecture, another important consideration is whether your application needs to handle more than text, such as images, audio, or video.

When building a GenAI system that handles images, audio, or documents, you have two main options. You can use a multimodal model that processes these inputs directly, or build a pipeline of specialized models that handles each step separately.

A voice assistant is a good example:

In the pipeline approach, speech is converted to text, processed by an LLM, and converted back to audio. This makes it easier to inspect intermediate results, apply safety checks, and debug problems. However, converting speech to text can lose information like tone, hesitation, and emphasis.

A native speech-to-speech model processes audio directly, preserving more of these details. It can also reduce latency, but debugging and applying text-based guardrails become harder if the model doesn't expose intermediate transcripts.

The same trade-off applies to document processing. For clean, text-heavy documents, extracting text and passing it to an LLM is often sufficient and inexpensive. But for complex tables, charts, handwriting, or unusual layouts, a vision-language model that examines the page directly may work better.

Here's how the approaches compare:

In practice, you don't have to choose just one approach.

For example, a voice assistant might use a native speech model for real-time conversations while generating transcripts separately for monitoring. Similarly, a document processing system might extract text from most pages and use a vision-language model only for complex layouts or poor-quality scans.

The right choice depends on the input, latency requirements, cost, and how much control you need over each processing step.

Multimodal inputs add their own processing and cost considerations. But even text-only applications can become expensive as input length grows. That brings us to another important factor in model selection: the context window.

Many current models support context windows of 128K, 200K, and even a million tokens or more. This raises an important question: if a model can process an entire document, do we still need retrieval?

Not always. Long context can be a simpler choice when the material fits within the model's context window, such as a contract, a technical manual, or a collection of source code files.

It's especially useful when the task requires understanding the entire document, like summarizing a report or finding inconsistencies across sections. For occasional requests, it may also be simpler and cheaper than building a retrieval pipeline.

However, retrieval-augmented generation (RAG) is often more practical for high-volume applications. Here's why.

Cost increases with input size. Suppose a model charges $3 per million input tokens. Sending 500,000 tokens costs $1.50 per question. With RAG, retrieving just 8,000 relevant tokens costs $0.024 in input tokens.

At 100,000 questions per day, that's $150,000 versus $2,400 in daily input token costs, excluding other prompt tokens, output tokens, and retrieval costs.

Longer inputs increase latency. Before generating a response, the model processes the input during the prefill phase. Processing hundreds of thousands of tokens requires more computation than processing a few thousand, which can increase the time to the first token.

More context doesn't always mean better answers. Models can struggle to find relevant information buried in long inputs, especially when surrounded by unrelated content. Just because a model supports one million tokens doesn't mean it performs equally well at every context length. You still need to evaluate accuracy on your actual tasks.

Large knowledge bases won't fit in context. A company's knowledge base might contain millions of documents with different access permissions and frequent updates. RAG can search the collection, filter results based on user permissions, and return only the relevant content. New information becomes searchable once the index is updated.

Here's a simple way to decide between the two approaches:

One factor that can change the cost comparison is prompt caching.

If many requests share the same large input, such as a product manual used by a support assistant, some providers can cache the shared prompt prefix and charge less when it's reused. This can make long-context approaches much more affordable.

However, caching is less useful when each request involves different documents or the shared content changes frequently.

In practice, long context and retrieval work well together. Retrieval finds the relevant documents, while a larger context window lets the model process more complete sections instead of relying on small, isolated chunks.

The best approach depends on your workload. In either case, give the model enough context to answer accurately without sending unnecessary information.

Most model selection decisions involve trade-offs between quality, latency, and cost. But data privacy and compliance requirements are often non-negotiable. They can rule out certain models or providers before you even compare their performance or pricing.

Ask about these constraints during requirements gathering.

Here are the most common ones:

Cloud platforms like AWS Bedrock, Google Vertex AI, and Azure offer another option. They provide managed model access with regional deployment options, enterprise security controls, and compliance agreements that may help meet requirements a direct provider API cannot.

Requirements can also vary within the same application.

For example, a customer support assistant might use a hosted API for general product questions but route requests containing sensitive account information to a self-hosted model.

However, this requires reliable data classification before routing, along with proper controls for logs, retrieval, and downstream services.

Apply compliance constraints at the start, before you compare models on quality or cost.

These requirements also make it valuable to keep your application independent of any single model or provider.

New models are released regularly, prices change, and providers retire older versions. The best model for your application today might not be the best choice six months from now.

To make switching easier, you can introduce an LLM gateway, a service that sits between your application and its models. Instead of calling providers directly, applications send requests to the gateway, which routes them to the appropriate model.

Instead of hardcoding model names, applications use logical names like support-answer or query-rewrite. The gateway maps each name to a specific model and version.

This lets you switch providers or upgrade models by changing the gateway configuration rather than modifying application code.

The gateway can also handle provider failover, rate limiting, per-customer quotas, logging, and token usage tracking. This keeps provider-specific logic in one place instead of spreading it across your applications.

What an LLM Gateway Doesn't Solve

An LLM gateway makes switching models easier, but it doesn't guarantee that different models will produce equivalent results.

There are three important things to consider:

That's why switching models should be treated like deploying a code change. Every new model should pass your evaluation suite before receiving production traffic.

One approach is to start with shadow mode, where the new model processes copies of real requests without returning its responses to users. This lets you compare quality, latency, and cost against the current model.

If the results are promising, gradually route a small percentage of live traffic to the new model. Monitor its performance and increase traffic as confidence grows.

Suppose you're designing a customer support assistant that handles 200,000 conversations per day. Most users ask about billing or product features, while some need help debugging complex integrations. The company also serves EU customers whose data must remain within the EU.

Here's how you might choose models for different tasks:

Rather than using one model for everything, each task gets a model suited to its quality, latency, cost, and compliance requirements.

All models sit behind an LLM gateway, allowing you to switch providers or upgrade individual models without changing application code. Before making a switch, you validate the new model against your evaluation suite to ensure it meets the same requirements.

Here's a quick summary of the main decisions:

Data privacy, residency, retention, and compliance requirements filter the options before any of these trade-offs apply.

Among the models that pass those filters, pick the cheapest and fastest one that meets your quality requirements on real evaluation data, and design the system so you can switch models when a better option arrives.