An LLM generates responses based on the information available in its context window. It doesn't automatically remember previous requests, access your database, or know anything about the user unless your system provides that information in the current request.

This makes the context window the main interface between your application and the model. For every request, your system must decide what to include: system instructions, tool definitions, conversation history, retrieved documents, and relevant memories from earlier sessions. Too little context can lead to incomplete or inaccurate answers. Too much increases cost and latency, and can reduce response quality.

Context engineering is about selecting, organizing, and managing this information so the model has what it needs to produce a useful response, even as conversations grow longer and agents run for hours.

In this chapter, we'll look at how to manage the context window as a limited budget, structure prompts for effective caching, handle long conversations, design long-term memory, and keep agents within their context limits. We'll also cover structured outputs that downstream systems can reliably process and the common failure modes you'll encounter in production.

Every LLM has a limited context window, such as 128K or 200K tokens, shared between the input and generated output. The input comes from several sources, all competing for the same space:

The context window is an upper limit, and there are three reasons to keep the input well below it.

Cost. Every input token adds to the cost of a request. In a chat application, conversation history is typically sent again with each new message, so the total cost grows faster than the conversation itself. For example, if each turn adds 1,000 tokens, the twentieth turn includes roughly 20,000 tokens of history. Across all 20 turns, that's 1,000 × (1 + 2 + ... + 20) = 210,000 input tokens, just for the growing history.

Latency. Longer inputs generally take more time to process before the model generates its first output token. A 100K-token prompt can have significantly higher prefill latency than a 5K-token prompt, especially when the input isn't cached.

Quality. More context doesn't always mean better answers. Old conversations, loosely related documents, and unnecessary tool definitions can distract the model from the information that matters.

A practical solution is to assign a token budget to each component and enforce those limits before every model call.

For example, suppose you're building a research assistant using a model with a 200K-token context window. Instead of using the entire window, you limit the input to 40K tokens to control cost and latency:

The six input components add up to 40K tokens, with another 4K reserved for the response.

The component responsible for managing these limits is typically called the context builder. It collects information from different sources, counts tokens, trims content that exceeds its budget, and assembles the final prompt in a consistent order.

For accurate token counts, use the model's tokenizer or the provider's token-counting API. The common estimate of four characters per token is useful for rough calculations with English text, but it becomes less reliable for code, JSON, and many non-English languages. Relying on it for hard limits can cause requests to exceed the context window unexpectedly.

Finally, the order of the context also matters. A consistent prompt structure improves caching efficiency, which can significantly reduce cost and latency for repeated requests.

Many LLM providers support prompt caching, which reuses previously computed results when requests share the same token prefix. Instead of processing those tokens again, the provider can serve them from cache. Cached input tokens often cost 10% to 50% of the regular input price, depending on the provider, and can reduce time to first token by skipping repeated computation.

The key detail is that caching depends on matching prefixes. If a token changes, the content after that point can no longer reuse the same cached prefix. This gives us a simple rule for structuring prompts: put stable content first and frequently changing content last.

Moving from left to right, the content changes more frequently. Tool definitions and system instructions typically remain the same across requests. User memories change occasionally, while conversation history grows with each turn. Retrieved documents and the current message are usually different for every request.

This ordering is particularly useful for multi-turn conversations. Since new messages are appended to the history, the previous turn's prompt can remain a valid prefix of the next request, allowing the provider to reuse more cached tokens.

It also affects where you place retrieved content. Adding documents near the end of the prompt, alongside the current message, preserves the existing prefix. Inserting them into the system prompt or earlier conversation history would break the cache match for everything that follows.

Several things can reduce cache effectiveness:

Most providers expose the number of cached input tokens in their API usage data. Monitor the proportion of eligible input tokens served from cache, along with the resulting cost and latency savings.

If cache usage drops suddenly after a deployment, check whether something near the beginning of the prompt has changed.

Conversation history grows with every turn, so eventually the system must decide what to keep and what to remove. There are three common approaches: sliding windows, token-based truncation, and summarization.

Sliding Window and Truncation

The simplest approach is to keep only the most recent messages. A sliding window retains the last N turns, while token-based truncation removes older messages until the history fits within its token budget.

Both approaches are simple and predictable, but a few details matter:

The main limitation is that discarded information is no longer available to the model. If the user shared an order number 40 turns ago and that message has been removed, the model won't know it unless the information is preserved elsewhere.

Summarization

Summarization preserves important information from older turns without keeping every message. When the history exceeds a threshold, a smaller, cheaper model summarizes the older conversation, while recent messages remain unchanged.

This gives the model a compressed version of the earlier conversation alongside the exact messages from recent turns, which are usually most relevant to the current request.

The challenge is deciding what the summary should preserve. A generic instruction like "Summarize this conversation" often produces a broad overview that misses important details. Instead, explicitly ask the summarizer to retain the user's goal, decisions made, specific values such as IDs and numbers, unresolved questions, and any commitments the assistant has made.

Summarization also has trade-offs. It's lossy, so details can disappear, and repeatedly summarizing previous summaries can introduce or amplify errors. It also requires an additional model call. To avoid adding latency to the main request, you can generate summaries asynchronously after responding to the user. Keep the original transcript in storage so earlier messages can still be retrieved when exact details are needed.

For short, task-focused conversations like customer support, a sliding window is often sufficient. Longer sessions benefit from summarization. A common setup combines both: a running summary of older turns, recent messages kept verbatim, and a hard token limit to prevent the context from growing too large.

These approaches manage history within a single conversation. But when a user returns next week and starts a new session, that history isn't automatically available. Preserving useful information across sessions requires long-term memory.

Long-term memory lets an assistant retain useful information across conversations, such as user preferences, facts they've shared, and outcomes of previous tasks. Designing a memory system involves four decisions: what to store, how to represent it, how to retrieve it, and how to keep it up to date.

What to Store

Memories generally fall into three categories:

A useful memory is something the user has explicitly shared, is likely to remain relevant, and could improve future responses. Most details from one-off tasks aren't worth storing, and the system shouldn't turn its own assumptions about the user into facts.

Credentials and other secrets should never be stored as memories. Sensitive personal information, such as health data, requires appropriate consent and stricter handling.

Extraction vs. Raw Storage

There are two main approaches to storing memories.

Raw storage preserves conversation messages, splits them into chunks, and creates embeddings so relevant excerpts can be retrieved later.

Extraction uses an LLM to identify useful facts and store them as short, standalone memories, such as "Prefers PostgreSQL over MySQL."

You can also combine both approaches. The original conversation serves as the source of truth, while extracted memories provide a compact, searchable representation. Each memory links back to the message it came from, making it easier to verify or correct later.

The Write and Read Paths

A memory system has two main workflows: a write path that creates and updates memories, and a read path that retrieves relevant memories for new requests.

On the write path, memory extraction typically runs as a background job after a conversation becomes idle, avoiding additional latency during normal responses.

The extractor identifies candidate facts, then compares them against existing memories. For each candidate, the system decides whether to add a new memory, update an existing one, delete an outdated memory, or skip a duplicate. Open-source memory frameworks like Mem0 use this add, update, delete, or skip approach.

The assistant can also expose a save_memory tool for explicit requests. For example, when a user says, "Remember that I'm on the platform team now," the system can save that information immediately rather than waiting for background extraction.

On the read path, a common approach is to retrieve memories in two layers:

Semantic similarity alone isn't always enough. An older memory might closely match the current query but no longer be accurate or useful. A better approach is to rank memories using a combination of relevance, recency, and importance. This approach was explored in the 2023 Generative Agents research from Stanford and Google.

Two implementation details are especially important.

First, enforce user isolation during retrieval. Apply user ID filters or tenant-level namespaces within the database query, along with authorization checks, rather than retrieving memories across users and filtering afterward. This prevents one user's information from leaking into another user's context.

Second, include timestamps with memories. For example, "As of March 2026, the user works on the payments team." This helps the model recognize information that might be outdated.

Memory retrieval also adds latency before response generation. To minimize this overhead, run memory lookups in parallel with document retrieval whenever possible.

Updates and Deletion

Memories can become outdated as the user's circumstances change.

Suppose a user moves from Bangalore to Pune. Simply keeping both locations creates a contradiction, while overwriting the original loses the history of the change.

A better approach is to mark the old memory as superseded using a valid_until timestamp and create a new record. Normal retrieval returns only the current information, while historical records can remain available for auditing when permitted.

Deletion requires more care. If a user asks the assistant to forget something or exercises a legal right to erasure under regulations like GDPR, removing the memory record alone isn't enough. The system must also account for other places where that information may exist:

To support deletion reliably, the system needs data lineage: a way to track which source messages produced each memory and where that information was subsequently copied or used.

Check how the vector database handles deletion. Some databases initially mark records as deleted and physically remove them during later compaction. The system must ensure deleted records become inaccessible immediately and are physically removed within the required retention period.

Long-term memory retains information across sessions. Agents that run for hours within a single session need a different technique to manage their context.

Agents accumulate context with every tool call and its result. A coding agent that reads files, runs tests, and searches documentation might add 5,000 to 20,000 tokens per step. After 30 or 40 steps, it can approach the limit of even a large context window, and response quality may decline well before that.

Several techniques help manage context growth, and production systems often combine them.

Tool result clearing. Tool outputs can account for a large portion of an agent's context. Once the agent has processed an output, the full content may no longer be necessary. Replacing it with a short summary, such as [run_tests output cleared: 3 failures in test_auth.py], frees up tokens while preserving the important result.

Offloading to storage. Large outputs, such as a 2,000-line log, can be stored in a file or object store. The context keeps only a reference and a short preview. If the agent needs the full output later, it can retrieve it through a tool.

Compaction. When context usage reaches a threshold, such as 70 to 80 percent of the allocated budget, the system asks a model to summarize the work completed so far. It then rebuilds the context using the system instructions, tool definitions, the summary, and the most recent steps.

Persistent notes. The agent maintains a progress file, such as a task checklist or notes document, outside the context window. These notes survive compaction and can be loaded again when the agent needs to recover important details or review its progress.

Sub-agents. Independent, context-heavy tasks can be delegated to sub-agents with their own context windows. For example, a sub-agent assigned to "Find every place this API is called" might inspect 50 files but return a concise 1,500-token report. The main agent receives the findings without carrying the entire search history.

Before compaction, much of the context consists of older tool outputs and intermediate steps the agent no longer needs. After compaction, the agent retains its instructions, a summary of the current state, and the recent steps relevant to its next action.

The summary needs to preserve the original goal and constraints, completed work, important findings, files or records modified, unresolved issues, and next steps. It should also record approaches that were tried and failed. Without this information, an agent may repeat the same unsuccessful attempts after compaction.

Compaction has trade-offs. It requires an additional model call, can reduce prompt cache reuse, and inevitably loses some detail. Rather than compacting on a fixed schedule, trigger it when context usage reaches a threshold or when accumulated content is no longer useful.

To evaluate the approach, test agents on long-running tasks that require multiple compactions. Measure task completion rates, repeated actions, and information lost across compactions, and compare the results against shorter tasks that don't require compaction.

So far, we've focused on managing what goes into the model. But when model responses are consumed by application code, the output also needs a reliable structure.

Many LLM responses in production systems are consumed by code rather than people. A classifier routes support tickets, an extractor fills database fields, and an agent generates arguments for tool calls. These components need predictable, machine-readable output that follows a defined schema.

There are four common approaches to generating structured output, each offering different levels of reliability:

Constrained decoding provides stronger guarantees by restricting which tokens the model can generate. The schema is converted into a grammar or equivalent decoding constraints. At each generation step, tokens that would violate the schema are excluded.

For example, if priority must be "low", "medium", or "high", the decoder prevents the model from generating any other value for that field.

Hosted providers offer this through strict structured output modes. Self-hosted inference engines like vLLM support constrained decoding through backends such as XGrammar and Outlines.

Here's a JSON schema for a support ticket classifier:

In this schema, order_id is required but can be null. Some providers require every property to appear in the required list when strict mode is enabled. Allowing null is a common way to represent a field that may not have a value.

Constrained decoding improves reliability, but it has several limitations:

For these reasons, structured output still requires application-level validation. A reliable pipeline checks three things: whether the response is valid JSON, whether it matches the expected schema, and whether the values satisfy business rules.

With strict constrained decoding, JSON parsing and schema validation should rarely fail for completed responses. Remaining failures then tend to come from business rules, such as an order ID that doesn't exist or a value outside an allowed business range.

For recoverable errors, include the validation failure in a retry so the model knows what to correct. However, don't let the model invent identifiers or other facts just to pass validation. Some failures should go directly to a fallback or human review.

Here's a simplified Python implementation using Pydantic. The call_llm and order_exists functions represent the model client and database lookup:

The example retries formatting and schema errors, but routes unverified order IDs to a fallback instead of asking the model to guess a replacement. A production implementation should also handle API errors, refusals, and incomplete responses explicitly.

Keep retries bounded, typically to one or two additional attempts. Repeated failures often indicate a problem that another attempt won't solve, while every retry adds latency and cost.

Finally, log validation failures and monitor the retry rate. A sudden increase can indicate changes in input data, model behavior, or a prompt update that introduced a regression.

Context-related problems rarely produce explicit errors. Instead, response quality gradually declines, making these issues harder to detect. Three failure modes are common.

Lost in the Middle

A 2023 study by Liu et al., Lost in the Middle, found that language models tend to use information near the beginning and end of a long context more reliably than information in the middle. When a relevant document was buried among 20 retrieved documents, answer accuracy dropped significantly.

Some newer long-context models reduce this effect, but performance can still decline as context grows, especially when relevant information is surrounded by unrelated content.

Several practices help reduce this problem:

Context Poisoning From Bad Memories

When incorrect information enters the context, the model may treat it as a reliable fact. Long-term memory makes this more dangerous because the error can persist across sessions.

An incorrect fact gets stored as memory, retrieved in a later conversation, and used to generate a response. If the memory system extracts that same fact from the new conversation, the error can reinforce itself.

Incorrect memories can originate in several ways:

The same issue can occur within a single agent session. If an incorrect conclusion gets included in a compaction summary, the agent may continue making decisions based on that mistake.

The main defense is to control what enters memory and how it is validated:

Prompt Bloat Over Time

System prompts tend to grow as applications evolve. Teams often add a rule after a production incident, such as "Never mention competitor pricing" or "Always confirm the order ID before issuing a refund."

Over time, a prompt that started at 800 tokens might grow to 6,000, with instructions that overlap, repeat, or contradict one another. Tool definitions can grow similarly, from 5 tools to 60.

This increases the cost of every request and makes it harder for the model to consistently follow individual instructions, especially when they compete for attention.

The solution is to treat prompts like code, with versioning, testing, and regular maintenance:

A practical compromise is to maintain a stable tool set for each agent type or task category. For example, a coding agent gets file-editing and testing tools, while a research agent gets search and retrieval tools. This keeps tool definitions manageable while preserving cache reuse within each category.

Let's see how these components work together when a user sends a message to a chat assistant with long-term memory.

When a new message arrives, the orchestrator retrieves relevant memories and documents in parallel. It also loads the conversation summary and recent messages, then assembles the context within the allocated token budget, placing stable content first to maximize prompt caching.

Once the model generates a response, it's streamed to the user. A background worker can then summarize older conversation history when needed and extract or update long-term memories. These operations run outside the response path, avoiding additional user-facing latency.

In a system design interview, start by explaining how the context is assembled, prioritized, and kept within budget. Then focus on the part most relevant to the system: conversation history for chat applications, long-term memory for personal assistants, context compaction for autonomous agents, or structured output for extraction and automation pipelines.

Give the model only the context it needs for the current task, keep the facts and decisions later turns depend on, and order the prompt so stable content can be cached.