When you call an LLM API with default settings, you let the model decide several important details: how much variation to allow, how long the answer can be, and when to stop. That is fine for a quick experiment. In a real application, you usually want more control.

LLM APIs expose parameters that shape generation: temperature, top_p, output-token limits, stop sequences, penalties, seeds, and log probabilities. These controls will not fix a poor prompt by themselves, but they help you manage cost, latency, variation, and output format.

In this chapter, you will learn what the most common parameters do, when to change them, and when to leave them alone.

Before tuning parameters, you need a simple model of how text generation works. An LLM does not "write" text the way a human does. It predicts one token at a time.

At each step, the model looks at everything generated so far and produces a probability distribution over all possible next tokens. The word "the" might have a 15% chance, "a" might have 8%, "Hello" might have 0.001%. The model then picks one token from this distribution, appends it to the output, and repeats.

The parameters we cover in this chapter all influence this token selection process. Some change the probability distribution itself. Others control when the process stops.

This loop runs hundreds or thousands of times for a single response. Each parameter below affects a different part of the loop.

Temperature is the parameter you will adjust most often. It controls how narrowly or broadly the model samples from the next-token probabilities.

How It Works

At each generation step, the model produces a probability distribution over possible next tokens. Temperature reshapes that distribution before a token is sampled.

Here is the math, simplified. The model produces raw scores (called logits) for each possible next token. These get converted to probabilities using the softmax function:

Here, T is the temperature. Different values produce different sampling behavior:

Code

Here is a quick way to see temperature in action. The script sends the same prompt at five different temperature values and prints the results:

Sample Output:

The pattern is what matters. At temperature 0, repeated runs tend to look very similar. At 0.7, the outputs vary but usually stay sensible. At 1.5, the model explores more unusual combinations, which can be useful for brainstorming but risky for production answers.

When to Use Each Temperature

Temperature is not the only way to control variation. top_p, also called nucleus sampling, limits which next tokens are eligible before the model samples one.

How It Works

Top-p works by sorting possible next tokens from most likely to least likely. The API then adds those probabilities until the total reaches your top_p value. Only tokens inside that group are eligible.

Say the model produces these probabilities for the next token:

With top_p = 0.8, the model only considers "the", "a", and "one" because their cumulative probability reaches 0.80. The remaining tokens are excluded. The model then samples from the reduced set.

With top_p = 0.95, the model considers the top 5 tokens. With top_p = 1.0 (the default), all tokens are eligible, so top_p has no effect.

Temperature vs Top-p

Both parameters control randomness, but they work differently:

In practice, adjust either temperature or top_p, not both at the same time unless you are deliberately experimenting. Changing both makes it harder to attribute changes in behavior. Most teams tune temperature first and leave top_p at 1.0.

Top-p dynamically adjusts how many tokens are eligible based on cumulative probability. Top-k takes a simpler approach: it keeps exactly the top K most probable tokens and discards the rest.

If top_k=50, the model only considers the 50 highest-probability tokens at each step, regardless of how much cumulative probability they cover. If top_k=1, generation becomes greedy: only the highest-ranked token is eligible.

Top K vs Top-p

Think of top_k as a fixed-size window and top_p as a variable-size window. top_p adapts to confidence: when the distribution is sharp, the nucleus may contain only a few tokens; when the distribution is flat, it may contain many. top_k always keeps the same number of candidates.

Code

Temperature and top_p introduce randomness by design, so the same prompt can produce different outputs on different runs. Reproducibility matters when you are testing a pipeline, debugging a prompt change, or showing a specific behavior to another engineer.

The seed parameter helps with that. It initializes the random number generator used during sampling, so repeated requests are more likely to follow the same path through the probability distributions.

How It Works

When you pass a seed value, the sampler uses it to initialize randomness. Same prompt, same seed, same model, and same parameters often produce the same output. It still depends on provider support and backend stability.

Sample Output:

Without the seed, temperature 0.7 would usually give you different names across runs. With seed=42, the runs are much more likely to match.

The Caveat: Best-Effort, Not Guaranteed

Seed-based reproducibility is best-effort across most providers. Even when a provider supports seeds, exact determinism is not guaranteed. Model serving infrastructure can change, and small numerical differences can affect a sampled token.

In practice, short outputs with the same model version are often reproducible. Long outputs or requests made weeks apart may differ. For testing and debugging, seed is useful because it reduces noise. For production logic, do not depend on exact string matching.

Some providers expose a system_fingerprint or similar backend identifier. If it changes, a seeded request may change too:

Every LLM has a context window: a fixed token budget shared by your input and the model's output. If you do not manage that budget, responses can be cut off or cost more than expected.

How Context Windows Work

The context window is the total number of tokens the model can handle in a single request. This includes everything: the system prompt, the conversation history, and the generated output.

Here is the relationship:

If your input uses 390,000 tokens of a 400K context window, the model has at most 10,000 tokens left for output, regardless of what output limit you request.

What max_completion_tokens Does

The max_completion_tokens parameter sets an upper limit on how many tokens the model can generate. It does not force the model to use the full amount. The model may stop earlier if it reaches a natural conclusion or hits a stop sequence.

You may still see older examples use max_tokens. On OpenRouter, both names refer to the same kind of output cap, but this course uses max_completion_tokens in new code.

When the model hits the output-token limit, it stops generating immediately, even if it is mid-sentence. The API response includes a finish_reason field that tells you why generation stopped:

Counting Tokens Across Providers

Tokens are not words. The word "understanding" is one word but may be split into multiple tokens depending on the tokenizer. Knowing how many tokens your input uses helps you set limits and predict costs.

You can use the tiktoken library to count tokens before making an API call. It runs offline and returns results instantly.

Running this will show you that common words often map to single tokens, while less common words get split into pieces. This is important because it means token count does not scale linearly with word count. A rough rule of thumb: 1 token is approximately 0.75 words in English, or about 4 characters.

Sometimes you need the model to stop at a specific point. You may want one short answer, or you may be generating text up to a delimiter such as Cons: or ### END. Stop sequences give you a simple way to do that.

How They Work

A stop sequence is a string that causes generation to stop when the model produces it. The stop sequence itself is usually not included in the returned text.

You can specify up to 4 stop sequences per request (depending on the API). The model stops as soon as it generates any one of them.

Practical Uses

Stop sequences are especially useful when you are generating structured or delimited content:

Another common pattern is using stop sequences with multi-part generation, where you want the model to generate content in sections:

LLMs sometimes repeat the same phrase, token, or idea more than you want, especially in longer outputs. Frequency and presence penalties give you two ways to reduce that behavior.

How They Differ

Both penalties lower the probability of tokens that have already appeared in the output, but they do it differently:

Frequency penalty (range: -2.0 to 2.0, default: 0) reduces the probability of a token proportionally to how many times it has already appeared. If the word "the" has appeared 5 times, it gets penalized 5 times as much as a word that appeared once.

Presence penalty (range: -2.0 to 2.0, default: 0) applies a flat penalty to any token that has appeared at all, regardless of how many times. Whether a word appeared once or fifty times, it gets the same penalty.

In short, frequency penalty helps with repeated wording, while presence penalty nudges the model toward covering new ground.

Code

With a high frequency penalty, the model usually uses a wider range of words and avoids repeating specific terms. With a high presence penalty, it tends to move to new topics more readily, which can help with coverage but may hurt coherence.

In practice, mild values such as 0.3 to 0.8 are often enough. Values above 1.5 can make the output feel forced or unnatural.

So far, every parameter we have covered changes how the model generates text. Logprobs do something different: they show the probability the model assigned to each output token, along with optional alternatives.

This is useful for debugging and evaluation. If a model classifies a support ticket as "billing", you can check whether "technical" was close behind. Logprobs give you token-level evidence about the alternatives the model considered.

How It Works

When you set logprobs=True, the API returns the log-probability of each token in the output. A log-probability is just the natural logarithm of the probability: a value of 0 means 100% confidence, and more negative values mean lower confidence. You can convert to a regular probability with exp(logprob).

You can also set top_logprobs (1 to 20) to see the probabilities of the top alternative tokens at each position.

For a simple factual prompt such as this, the chosen token will usually have very high probability. That is useful signal, but it is not a general proof of factual correctness.

Practical Uses

Logprobs are most useful when the model is less certain:

On an input like this, the top class often comes back with middling confidence, with a close runner-up not far behind. That spread tells you the input is ambiguous enough to justify human review or a separate "mixed" category.

Here are the most common uses for logprobs:

LLM APIs charge per token. Input tokens and output tokens often have different prices. If you are building a product, you need to estimate costs before traffic grows. A chatbot that handles thousands of conversations a day can become expensive if you never measure usage.

How Pricing Works

Most providers charge per million tokens, with separate rates for input and output. Output tokens are often more expensive because the model has to generate them one step at a time.

Use current provider pricing when estimating production cost. Prices change, and gateway pricing may differ from direct-provider pricing.

Building a Cost Calculator

Here is a small utility that estimates the cost of a request before you make it:

One caveat: tokenizers differ across model families. This example uses OpenAI's o200k_base tokenizer because the course model is an OpenAI model. For other providers, use the provider's tokenizer when available. Treat the numbers from response.usage as the source of truth for actual billing.

Tracking Actual Costs

Every API response includes a usage field with the actual token counts. Use this to track real costs:

For production applications, wrap this in a logging function that tracks costs across all API calls:

Here is a quick reference for the parameters covered in this chapter:

Common Parameter Recipes

These combinations work well for specific use cases:

Now let's tie the ideas together. This script sends the same prompt at different temperature values and measures how much the outputs vary.

When you run this, expect to see something like:

The takeaway is that temperature changes how much variation you should expect from the same prompt, but it is only one part of the system. Prompt specificity, schema constraints, model choice, and post-processing matter just as much in production.