{"title":"11.6 Prompt Caching","description":"","content":"# 11.6 Prompt Caching\n\n## The Problem\n\nYou have spent the last few chapters making your prompts leaner (11.2), routing to cheaper models (11.3), and squeezing out latency (11.4). But there is a cost you have not eliminated yet: the cost of processing the same tokens over and over again.\n\nThink about what happens on every API call. Your system prompt, your few-shot examples, your tool definitions, they all get sent to the model. The model reads them, computes internal representations for every single token through every attention layer, and then finally gets to the part that actually changes: the user's message. If your static prefix is 3,000 tokens and you handle 50,000 requests a day, you are asking the model to re-process 150 million tokens of content it has already seen before. That is not just expensive. It is wasteful.\n\nIn 11.2, we introduced prompt caching as one of several optimization techniques. This chapter goes much deeper. We will look at how caching works under the hood, walk through the APIs for Anthropic, OpenAI, and Google Gemini, and build a systematic approach to structuring prompts for maximum cache hits. By the end, you will know how to measure cache performance and calculate exactly how much money caching is saving you.\n\n**What you will learn:**\n\n- What happens inside the model that makes prompt caching possible\n- How to use Anthropic's explicit caching with `cache_control`\n- How OpenAI's automatic caching works and how to take advantage of it\n- How Google Gemini's context caching handles large document contexts\n- How to structure prompts so the cacheable prefix is as large as possible\n- How to measure cache hit rates and calculate real cost savings\n- Common mistakes that silently destroy your cache hit rate\n\n---\n\n## How Prompt Caching Works Under the Hood\n\nTo understand why prompt caching is such a big deal, you need to understand what the model actually does when it processes your prompt. This is where the connection to the next chapter (11.7 KV Cache) starts.\n\nWhen you send a prompt to a model, every token passes through dozens of attention layers. At each layer, the model computes two matrices for each token: a **key** matrix and a **value** matrix (the \"KV\" in KV cache). These matrices encode how that token relates to every other token in the sequence. For a model like Claude Sonnet with many layers, this computation is the single most expensive part of processing your request.\n\nHere is the critical insight: if the first 3,000 tokens of your prompt are identical to a previous request, the key-value matrices for those tokens are also identical. There is no reason to recompute them. Prompt caching stores these precomputed KV matrices on the provider's infrastructure so that subsequent requests with the same prefix can skip straight to computing the new tokens.\n\n\n```mermaid\nflowchart TD\n subgraph first[\"First Request (Cache Miss)\"]\n direction TB\n P1[\"Full Prompt
(system + examples + query)\"]:::primary\n KV1[\"Compute KV matrices
for ALL tokens\"]:::orange\n STORE[\"Store prefix KV matrices
in cache\"]:::teal\n GEN1[\"Generate response\"]:::green\n P1 --> KV1 --> STORE --> GEN1\n end\n\n subgraph second[\"Subsequent Request (Cache Hit)\"]\n direction TB\n P2[\"Same prefix +
new query\"]:::primary\n LOAD[\"Load cached KV matrices
for prefix (skip computation)\"]:::teal\n KV2[\"Compute KV matrices
only for new tokens\"]:::orange\n GEN2[\"Generate response\"]:::green\n P2 --> LOAD --> KV2 --> GEN2\n end\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe left side shows what happens on the first request: the model computes KV matrices for every token, stores the prefix portion in cache, and generates the response. The right side shows a subsequent request with the same prefix: the model loads the cached KV matrices, only computes the new tokens, and generates the response. The savings come from skipping that expensive KV computation for the entire prefix.\n\n### Prefix Matching\n\nThe matching mechanism is strict. Caching works on the **prefix** of your prompt, meaning it matches from the very first token forward. If the first 2,500 tokens match a cached entry but token 2,501 differs, you get a cache hit for those 2,500 tokens and a miss for everything after. If token 1 differs, you get no cache hit at all.\n\nThis is not fuzzy matching. It is byte-for-byte comparison. Even a single whitespace change, a different newline character, or a reordered sentence in your system prompt will break the match. This strict matching is what makes the prompt structuring advice later in this chapter so important.\n\n---\n\n## Anthropic's Prompt Caching\n\nAnthropic takes an explicit approach to caching. You tell the API exactly which parts of your prompt should be cached by adding a `cache_control` parameter to message blocks. This gives you fine-grained control over what gets cached and makes it easy to reason about cache behavior.\n\n### The Economics\n\n\n| Metric | Value |\n|--------|-------|\n| Cache write cost | 25% more than base input price |\n| Cache read cost | 90% less than base input price |\n| Minimum cacheable prefix | 1,024 tokens |\n| TTL (time to live) | 5 minutes, resets on each hit |\n\n\nThe math works out in your favor quickly. Suppose your prefix is 3,000 tokens using Claude Sonnet. The first request pays a 25% surcharge on those 3,000 tokens to write the cache. Every subsequent request within 5 minutes pays only 10% of the normal price for those tokens. After just two cache hits, you have already recouped the write cost and started saving.\n\nThe 5-minute TTL with reset-on-hit is a nice design choice. As long as your application has steady traffic (at least one request every 5 minutes with the same prefix), the cache stays warm indefinitely. It only expires during quiet periods.\n\n### Code Example\n\n\n```python\nimport anthropic\n\nclient = anthropic.Anthropic()\n\n# Define the static parts of your prompt that should be cached.\n# These will be identical across all requests.\nSYSTEM_BLOCKS = [\n {\n \"type\": \"text\",\n \"text\": (\n \"You are a senior financial analyst assistant. You help users analyze \"\n \"financial statements, calculate key metrics, and identify trends.\\n\\n\"\n \"Guidelines:\\n\"\n \"- Always show your calculations step by step\\n\"\n \"- Flag any numbers that look unusual or inconsistent\\n\"\n \"- Use standard financial terminology\\n\"\n \"- When uncertain, state your confidence level\\n\"\n \"- Format large numbers with commas for readability\"\n ),\n },\n {\n \"type\": \"text\",\n \"text\": (\n \"Reference data:\\n\\n\"\n \"Industry average P/E ratios (2024):\\n\"\n \"- Technology: 28.5\\n\"\n \"- Healthcare: 22.3\\n\"\n \"- Financial Services: 14.8\\n\"\n \"- Consumer Discretionary: 25.1\\n\"\n \"- Energy: 11.2\\n\"\n \"- Industrials: 20.7\\n\\n\"\n \"Key formulas:\\n\"\n \"- P/E Ratio = Stock Price / Earnings Per Share\\n\"\n \"- ROE = Net Income / Shareholder Equity\\n\"\n \"- Debt-to-Equity = Total Liabilities / Shareholder Equity\\n\"\n \"- Current Ratio = Current Assets / Current Liabilities\\n\"\n \"- Quick Ratio = (Current Assets - Inventory) / Current Liabilities\\n\"\n \"- EBITDA Margin = EBITDA / Revenue\\n\"\n # ... imagine 50+ more lines of reference data and examples\n # to reach the 1,024 token minimum\n ),\n \"cache_control\": {\"type\": \"ephemeral\"}, # Cache everything up to here\n },\n]\n\ndef analyze_financials(user_query: str) -> dict:\n \"\"\"Send a financial analysis query with cached system context.\"\"\"\n response = client.messages.create(\n model=\"claude-sonnet-4-20250514\",\n max_tokens=2048,\n system=SYSTEM_BLOCKS,\n messages=[\n {\"role\": \"user\", \"content\": user_query}\n ],\n )\n\n # Extract cache performance metrics from the response\n usage = response.usage\n cache_read = getattr(usage, \"cache_read_input_tokens\", 0)\n cache_write = getattr(usage, \"cache_creation_input_tokens\", 0)\n regular_input = usage.input_tokens\n\n print(f\"Input tokens: {regular_input}\")\n print(f\"Cache read tokens: {cache_read}\")\n print(f\"Cache write tokens: {cache_write}\")\n\n if cache_read > 0:\n print(\"Cache HIT - saved ~90% on prefix tokens\")\n elif cache_write > 0:\n print(\"Cache MISS (first request) - prefix written to cache\")\n\n return {\n \"response\": response.content[0].text,\n \"cache_hit\": cache_read > 0,\n \"cache_read_tokens\": cache_read,\n \"cache_write_tokens\": cache_write,\n }\n\n# First request: cache miss, writes the prefix to cache\nresult1 = analyze_financials(\"What's Apple's P/E ratio compared to industry average?\")\n\n# Second request: cache hit, skips prefix computation\nresult2 = analyze_financials(\"Calculate the debt-to-equity ratio for this balance sheet...\")\n```\n\n\nA few things to notice in this code. The `cache_control` parameter sits on the second system block, which means everything from the start of the prompt through that block gets cached as a single prefix. The user message is outside the cached region, so it can change freely without affecting cache hits. And the response's `usage` object tells you exactly what happened: `cache_read_input_tokens` is nonzero on a hit, `cache_creation_input_tokens` is nonzero on a miss.\n\n### Where to Place cache_control\n\nYou can place `cache_control` on up to four content blocks. Each one marks the end of a cacheable prefix. This gives you a layered caching strategy:\n\n\n```python\nsystem = [\n {\n \"type\": \"text\",\n \"text\": \"Core system instructions...\",\n \"cache_control\": {\"type\": \"ephemeral\"}, # Breakpoint 1\n },\n {\n \"type\": \"text\",\n \"text\": \"Tool definitions (JSON schema for 10 tools)...\",\n \"cache_control\": {\"type\": \"ephemeral\"}, # Breakpoint 2\n },\n {\n \"type\": \"text\",\n \"text\": \"Few-shot examples...\",\n \"cache_control\": {\"type\": \"ephemeral\"}, # Breakpoint 3\n },\n]\n```\n\n\nIf all three blocks match a cached prefix, you get the full cache benefit. If only the first two match (because you changed the few-shot examples), you still get a partial cache hit on the system instructions and tool definitions. This layered approach is useful when parts of your prefix change at different frequencies.\n\n---\n\n## OpenAI's Prompt Caching\n\nOpenAI takes the opposite design philosophy: caching is fully automatic. You do not mark anything, you do not opt in, and there is no write cost. The system detects repeated prefixes of 1,024 tokens or more and caches them transparently.\n\n### The Economics\n\n\n| Metric | Value |\n|--------|-------|\n| Cache write cost | None (free) |\n| Cache read cost | 50% less than base input price |\n| Minimum cacheable prefix | 1,024 tokens |\n| TTL | 5-10 minutes (not officially documented) |\n| Scope | Shared across requests within the same organization |\n\n\nThe 50% discount is less aggressive than Anthropic's 90%, but the zero write cost means you start saving from the very first cache hit. There is no \"break even\" calculation needed. Every hit saves money from the start.\n\nOne important detail: OpenAI caches based on the **longest matching prefix**. If your prompt is 4,000 tokens and the first 3,500 match a cached entry, you get a cache hit on 3,500 tokens and pay full price only for the remaining 500. The matching happens in 128-token increments, so the cached portion will be a multiple of 128 tokens.\n\n### Code Example\n\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI()\n\n# Define a substantial system prompt that exceeds 1,024 tokens.\n# OpenAI caches this automatically - no opt-in needed.\nSYSTEM_PROMPT = \"\"\"You are a senior financial analyst assistant. You help users analyze\nfinancial statements, calculate key metrics, and identify trends.\n\nGuidelines:\n- Always show your calculations step by step\n- Flag any numbers that look unusual or inconsistent\n- Use standard financial terminology\n- When uncertain, state your confidence level\n- Format large numbers with commas for readability\n\nReference data:\n\nIndustry average P/E ratios (2024):\n- Technology: 28.5\n- Healthcare: 22.3\n- Financial Services: 14.8\n- Consumer Discretionary: 25.1\n- Energy: 11.2\n- Industrials: 20.7\n\nKey formulas:\n- P/E Ratio = Stock Price / Earnings Per Share\n- ROE = Net Income / Shareholder Equity\n- Debt-to-Equity = Total Liabilities / Shareholder Equity\n- Current Ratio = Current Assets / Current Liabilities\n- Quick Ratio = (Current Assets - Inventory) / Current Liabilities\n- EBITDA Margin = EBITDA / Revenue\n- Free Cash Flow = Operating Cash Flow - Capital Expenditures\n- Gross Margin = (Revenue - COGS) / Revenue\n- Operating Margin = Operating Income / Revenue\n- Net Margin = Net Income / Revenue\n\nValuation approaches:\n- Comparable Company Analysis: Use P/E, EV/EBITDA, P/S ratios\n- Discounted Cash Flow: Project 5-year FCF, apply terminal value\n- Precedent Transactions: Look at similar M&A deals\n\n... (additional reference data, examples, and instructions to exceed 1,024 tokens)\n\"\"\"\n\n# Tool definitions also contribute to the cached prefix\nTOOLS = [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"get_stock_price\",\n \"description\": \"Get current stock price and basic stats\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"ticker\": {\"type\": \"string\", \"description\": \"Stock ticker symbol\"}\n },\n \"required\": [\"ticker\"],\n },\n },\n },\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"get_financial_statement\",\n \"description\": \"Get income statement, balance sheet, or cash flow\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"ticker\": {\"type\": \"string\"},\n \"statement_type\": {\n \"type\": \"string\",\n \"enum\": [\"income\", \"balance_sheet\", \"cash_flow\"],\n },\n \"period\": {\"type\": \"string\", \"enum\": [\"annual\", \"quarterly\"]},\n },\n \"required\": [\"ticker\", \"statement_type\"],\n },\n },\n },\n]\n\ndef analyze_financials(user_query: str) -> dict:\n \"\"\"Send a query with automatic prefix caching.\"\"\"\n response = client.chat.completions.create(\n model=\"gpt-4o\",\n messages=[\n {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": user_query},\n ],\n tools=TOOLS,\n )\n\n # Check cache stats in usage details\n usage = response.usage\n cached_tokens = 0\n\n if hasattr(usage, \"prompt_tokens_details\") and usage.prompt_tokens_details:\n cached_tokens = getattr(usage.prompt_tokens_details, \"cached_tokens\", 0)\n\n print(f\"Prompt tokens: {usage.prompt_tokens}\")\n print(f\"Cached tokens: {cached_tokens}\")\n print(f\"New tokens: {usage.prompt_tokens - cached_tokens}\")\n\n if cached_tokens > 0:\n savings_pct = (cached_tokens / usage.prompt_tokens) * 50 # 50% discount\n print(f\"Input cost reduction: ~{savings_pct:.1f}%\")\n\n return {\n \"response\": response.choices[0].message,\n \"cached_tokens\": cached_tokens,\n \"total_prompt_tokens\": usage.prompt_tokens,\n }\n\n# Both calls use the same system prompt and tools.\n# The second call should show cached_tokens > 0.\nresult1 = analyze_financials(\"Analyze Apple's P/E ratio vs the tech industry average.\")\nresult2 = analyze_financials(\"What does Microsoft's free cash flow trend look like?\")\n```\n\n\nSince OpenAI caching is automatic, the key discipline is keeping your prompt prefix consistent. If you dynamically generate any part of the system prompt (inserting a timestamp, shuffling few-shot examples, adding a request ID), you will break the prefix match without realizing it. We will cover this in the pitfalls section.\n\n---\n\n## Google Gemini Context Caching\n\nGoogle takes a fundamentally different approach. Instead of caching prompt prefixes implicitly, you explicitly create a **cached content** object via the API, get back a reference ID, and then use that reference in subsequent requests. This design is particularly useful for very large contexts, like feeding an entire codebase, a long document, or a book into the model.\n\n### The Economics\n\n\n| Metric | Value |\n|--------|-------|\n| Cache write cost | Standard input price (charged once) |\n| Cache read cost | 75% less than base input price |\n| Minimum cacheable content | 32,768 tokens |\n| TTL | Configurable (default 1 hour, max 24 hours) |\n| Storage cost | Charged per hour while cache is alive |\n\n\nThe 32,768-token minimum is much higher than Anthropic or OpenAI, which means Gemini context caching is not useful for short system prompts. It is designed for scenarios where you are working with large documents, full codebases, or extensive reference materials that you want to query multiple times.\n\n### Code Example\n\n\n```python\nfrom google import genai\nfrom google.genai import types\n\nclient = genai.Client()\n\n# Step 1: Create a cached content object with your large context.\n# This is charged once at the standard input rate.\nlarge_document = open(\"company_annual_report_2024.txt\").read() # ~50,000 tokens\n\ncached_content = client.caches.create(\n model=\"gemini-2.0-flash-001\",\n config=types.CreateCachedContentConfig(\n system_instruction=(\n \"You are a financial analyst. Answer questions about the \"\n \"provided annual report. Cite specific page numbers and sections.\"\n ),\n contents=[\n types.Content(\n role=\"user\",\n parts=[types.Part(text=large_document)],\n )\n ],\n ttl=\"3600s\", # Keep cache alive for 1 hour\n display_name=\"annual-report-2024\",\n ),\n)\n\nprint(f\"Cache created: {cached_content.name}\")\nprint(f\"Token count: {cached_content.usage_metadata.total_token_count}\")\nprint(f\"Expires: {cached_content.expire_time}\")\n\n# Step 2: Use the cached content in multiple requests.\n# Each request pays 75% less for the cached document tokens.\ndef query_report(question: str) -> str:\n \"\"\"Ask a question about the cached annual report.\"\"\"\n response = client.models.generate_content(\n model=\"gemini-2.0-flash-001\",\n contents=question,\n config=types.GenerateContentConfig(\n cached_content=cached_content.name,\n ),\n )\n return response.text\n\n# All these queries reference the same cached document\nanswer1 = query_report(\"What was total revenue for fiscal year 2024?\")\nanswer2 = query_report(\"Summarize the risk factors section.\")\nanswer3 = query_report(\"How did operating margins change year over year?\")\n\n# Step 3: When done, delete the cache to stop storage charges.\nclient.caches.delete(name=cached_content.name)\n```\n\n\nThe workflow is different from Anthropic and OpenAI. You are managing a cache lifecycle explicitly: create it, use it across multiple requests, and delete it when done. This makes Gemini context caching feel more like a database than a transparent optimization. The upside is predictability. You know exactly what is cached, how long it lives, and what it costs. The downside is the operational overhead of managing cache objects.\n\n### When to Use Gemini Context Caching\n\nGemini's approach shines in specific scenarios:\n\n- **Document QA:** Load a 100-page report once, ask 50 questions about it\n- **Codebase analysis:** Cache an entire repository, then run multiple analysis queries\n- **Long conversation contexts:** Cache a conversation history that multiple agents need to reference\n- **Batch processing:** Cache a reference dataset, then classify thousands of items against it\n\nIt is less useful for typical chatbot scenarios where your system prompt is only 1,000-2,000 tokens. The 32,768-token minimum means your cacheable content needs to be substantial.\n\n---\n\n## Structuring Prompts for Maximum Cache Hits\n\nThis is where the practical payoff lives. You can have the perfect caching API calls, but if your prompt is structured poorly, your cache hit rate will be close to zero. The golden rule is simple: **static content first, dynamic content last.**\n\n### The Optimal Prompt Layout\n\n\n```mermaid\nflowchart TD\n subgraph cached[\"Cacheable Zone (static across requests)\"]\n direction TB\n SYS[\"1. System Instructions
(role, guidelines, rules)\"]:::primary\n TOOLS[\"2. Tool Definitions
(function schemas)\"]:::primary\n FEW[\"3. Few-Shot Examples
(fixed set)\"]:::primary\n end\n\n subgraph dynamic[\"Dynamic Zone (changes per request)\"]\n direction TB\n CTX[\"4. Retrieved Context
(RAG results)\"]:::orange\n HIST[\"5. Conversation History
(prior turns)\"]:::orange\n USER[\"6. User Message
(current query)\"]:::orange\n end\n\n SYS --> TOOLS --> FEW --> CTX --> HIST --> USER\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\nEverything in the cacheable zone is identical across requests. The boundary between zones is where the cache match ends and fresh computation begins. The larger the cacheable zone, the more you save.\n\n### Before and After: A Cache-Optimized Prompt\n\nHere is what a poorly structured prompt looks like versus a cache-optimized one.\n\n**Before (poor cache behavior):**\n\n\n```python\n# BAD: Dynamic content mixed into the prefix\nmessages = [\n {\n \"role\": \"system\",\n \"content\": (\n f\"You are a support agent. Current time: {datetime.now().isoformat()}\\n\" # Breaks cache every second\n f\"Request ID: {uuid4()}\\n\" # Breaks cache every request\n \"Be helpful and professional.\\n\"\n \"Tools available: search_docs, create_ticket, get_order_status\\n\"\n ),\n },\n {\"role\": \"user\", \"content\": user_query},\n]\n```\n\n\nThis prompt will **never** get a cache hit. The timestamp changes every second and the request ID changes every call, so the prefix is different every time. Every token after the first difference is recomputed from scratch.\n\n**After (cache-optimized):**\n\n\n```python\n# GOOD: Static content first, dynamic content at the end\nmessages = [\n {\n \"role\": \"system\",\n \"content\": (\n \"You are a support agent. Be helpful and professional.\\n\"\n \"Tools available: search_docs, create_ticket, get_order_status\\n\"\n # All static content stays in the system prompt\n ),\n },\n # Dynamic metadata goes in the user message, after the cached prefix\n {\n \"role\": \"user\",\n \"content\": (\n f\"[Context: time={datetime.now().isoformat()}, \"\n f\"request_id={uuid4()}]\\n\\n\"\n f\"{user_query}\"\n ),\n },\n]\n```\n\n\nSame information, completely different cache behavior. The system prompt is now frozen. The timestamp and request ID moved into the user message, which is outside the cached prefix. Every request with this template gets a cache hit on the system prompt.\n\n### Few-Shot Example Ordering\n\nThis catches people off guard. If you have 5 few-shot examples and you randomly shuffle them per request (maybe thinking it helps the model generalize), you will destroy your cache hit rate. The cache requires byte-identical prefixes, so randomizing example order means every permutation creates a separate cache entry.\n\nThe fix is straightforward: pick a fixed order for your few-shot examples and freeze it. If you need variety, create a few fixed sets (Set A, Set B, Set C) and route requests consistently to the same set. This way, each set maintains its own cached prefix.\n\n\n```python\n# BAD: Shuffling examples per request\nimport random\n\ndef build_prompt(examples, user_query):\n random.shuffle(examples) # Different order = different prefix = cache miss\n examples_text = \"\\n\\n\".join(examples)\n return f\"Examples:\\n{examples_text}\\n\\nQuery: {user_query}\"\n\n# GOOD: Fixed example order\nFROZEN_EXAMPLES = [\n \"Example 1: How do I reset my password? -> Go to /reset...\",\n \"Example 2: What's the pricing? -> See techcorp.com/pricing...\",\n \"Example 3: My order hasn't arrived -> Check order status at...\",\n]\nFROZEN_EXAMPLES_TEXT = \"\\n\\n\".join(FROZEN_EXAMPLES)\n\ndef build_prompt(user_query):\n return f\"Examples:\\n{FROZEN_EXAMPLES_TEXT}\\n\\nQuery: {user_query}\"\n```\n\n\n---\n\n## Measuring Cache Performance\n\nYou are using caching. But is it working? Without measurement, you are flying blind. A single code change somewhere upstream could silently break your prefix consistency and tank your cache hit rate overnight.\n\n### A Cache Performance Tracker\n\nHere is a practical tracker that logs every request's cache behavior and calculates rolling statistics:\n\n\n```python\nimport time\nfrom dataclasses import dataclass, field\n\n@dataclass\nclass CacheStats:\n \"\"\"Track prompt cache performance over time.\"\"\"\n\n hits: int = 0\n misses: int = 0\n total_cached_tokens: int = 0\n total_input_tokens: int = 0\n cost_without_cache: float = 0.0\n cost_with_cache: float = 0.0\n history: list = field(default_factory=list)\n\n def record(self, input_tokens: int, cached_tokens: int, price_per_million: float):\n \"\"\"Record a single request's cache performance.\"\"\"\n is_hit = cached_tokens > 0\n if is_hit:\n self.hits += 1\n else:\n self.misses += 1\n\n self.total_cached_tokens += cached_tokens\n self.total_input_tokens += input_tokens\n\n # Calculate costs\n uncached_tokens = input_tokens - cached_tokens\n full_cost = input_tokens * price_per_million / 1_000_000\n actual_cost = (\n uncached_tokens * price_per_million / 1_000_000\n + cached_tokens * price_per_million * 0.1 / 1_000_000 # 90% discount\n )\n\n self.cost_without_cache += full_cost\n self.cost_with_cache += actual_cost\n\n self.history.append({\n \"timestamp\": time.time(),\n \"input_tokens\": input_tokens,\n \"cached_tokens\": cached_tokens,\n \"hit\": is_hit,\n })\n\n @property\n def hit_rate(self) -> float:\n total = self.hits + self.misses\n return self.hits / total if total > 0 else 0.0\n\n @property\n def token_cache_rate(self) -> float:\n if self.total_input_tokens == 0:\n return 0.0\n return self.total_cached_tokens / self.total_input_tokens\n\n @property\n def total_savings(self) -> float:\n return self.cost_without_cache - self.cost_with_cache\n\n def report(self):\n \"\"\"Print a summary of cache performance.\"\"\"\n total_requests = self.hits + self.misses\n print(f\"Cache Performance Report\")\n print(f\"{'='*45}\")\n print(f\"Total requests: {total_requests:,}\")\n print(f\"Cache hits: {self.hits:,}\")\n print(f\"Cache misses: {self.misses:,}\")\n print(f\"Hit rate: {self.hit_rate:.1%}\")\n print(f\"Token cache rate: {self.token_cache_rate:.1%}\")\n print(f\"{'='*45}\")\n print(f\"Cost without cache: ${self.cost_without_cache:.4f}\")\n print(f\"Cost with cache: ${self.cost_with_cache:.4f}\")\n print(f\"Total savings: ${self.total_savings:.4f}\")\n if self.cost_without_cache > 0:\n pct = (self.total_savings / self.cost_without_cache) * 100\n print(f\"Savings percentage: {pct:.1f}%\")\n\n# Usage with Anthropic\nstats = CacheStats()\n\nfor query in user_queries:\n response = client.messages.create(\n model=\"claude-sonnet-4-20250514\",\n max_tokens=1024,\n system=SYSTEM_BLOCKS, # From earlier example\n messages=[{\"role\": \"user\", \"content\": query}],\n )\n\n usage = response.usage\n cache_read = getattr(usage, \"cache_read_input_tokens\", 0)\n stats.record(\n input_tokens=usage.input_tokens + cache_read,\n cached_tokens=cache_read,\n price_per_million=3.0, # Claude Sonnet input price\n )\n\nstats.report()\n```\n\n\n**Output:**\n\n\n```shell\nCache Performance Report\n=============================================\nTotal requests: 1,000\nCache hits: 987\nCache misses: 13\nHit rate: 98.7%\nToken cache rate: 72.4%\n=============================================\nCost without cache: $9.6000\nCost with cache: $3.9360\nTotal savings: $5.6640\nSavings percentage: 59.0%\n```\n\n\nTwo metrics matter most here. **Hit rate** tells you how many of your requests are benefiting from the cache. If this is below 90%, something is wrong with your prefix consistency. **Token cache rate** tells you what fraction of your total input tokens are being served from cache. This number depends on the ratio of your static prefix to your dynamic content. A high hit rate with a low token cache rate means your prefix is small relative to the dynamic content, which is fine, it just means the savings per request are modest.\n\n### What Good Numbers Look Like\n\n\n| Metric | Poor | Acceptable | Good | Excellent |\n|--------|------|------------|------|-----------|\n| Hit rate | < 50% | 50-80% | 80-95% | > 95% |\n| Token cache rate | < 20% | 20-40% | 40-60% | > 60% |\n| Cost savings | < 10% | 10-30% | 30-50% | > 50% |\n\n\nIf your hit rate is below 50%, you almost certainly have dynamic content leaking into your prefix. Check for timestamps, UUIDs, rotating example sets, or dynamically generated instructions.\n\n---\n\n## Provider Comparison\n\nHere is the full comparison across the three major providers:\n\n\n| Feature | Anthropic | OpenAI | Google Gemini |\n|---------|-----------|--------|---------------|\n| **Opt-in mechanism** | Explicit (`cache_control`) | Automatic | Explicit (create cache object) |\n| **Minimum prefix** | 1,024 tokens | 1,024 tokens | 32,768 tokens |\n| **Cache read discount** | 90% off | 50% off | 75% off |\n| **Cache write cost** | 25% surcharge | None | Standard input price |\n| **TTL** | 5 min (resets on hit) | ~5-10 min | Configurable (up to 24h) |\n| **Matching granularity** | Byte-exact prefix | 128-token chunks | Explicit reference ID |\n| **Scope** | Per API key | Per organization | Per project |\n| **Best for** | Medium prefixes, high control | Any workload, zero effort | Large documents, long contexts |\n\n\nWhich one should you use? It depends on your situation:\n\n- **Anthropic** gives you the most control and the deepest discount (90%). If you are already using Claude and want to squeeze maximum savings from a well-structured prefix, this is the best option.\n- **OpenAI** is the easiest to adopt because there is nothing to change. If you have consistent prompt prefixes over 1,024 tokens, you are already getting cached automatically. The 50% discount is lower, but the zero-effort setup makes up for it.\n- **Google Gemini** is the right choice when you are working with very large contexts. If your use case involves loading a 100-page document and asking many questions about it, the explicit cache management and 75% discount make it cost-effective despite the 32K minimum.\n\n---\n\n## Common Pitfalls\n\nAfter working through the APIs and prompt structuring, here are the mistakes that trip up teams most often. These are the silent cache killers, the things that make your cache hit rate plummet without any error messages.\n\n### 1. Dynamic Content in the Prefix\n\nThis is the most common one, and we touched on it earlier. Timestamps, request IDs, session tokens, or any per-request metadata in your system prompt will destroy cache hits.\n\n\n```python\n# Subtle bug: date changes daily, breaking cache every midnight\nsystem_prompt = f\"You are a helpful assistant. Today's date is {date.today()}. ...\"\n```\n\n\nMove all dynamic values into the user message or into a separate message block after the cached prefix.\n\n### 2. Inconsistent Prompt Templates Across Code Paths\n\nYour application might have multiple places that construct the system prompt: one for the API endpoint, one for the background worker, one for the admin tool. If they format the prompt slightly differently (extra newline, different whitespace, slightly different wording), each code path creates its own cache entry instead of sharing one.\n\nThe fix: define your system prompt in exactly one place and import it everywhere.\n\n\n```python\n# prompts.py - single source of truth\nSUPPORT_SYSTEM_PROMPT = \"\"\"You are a support agent. Be helpful and professional.\n\nRules:\n- Greet by name if provided\n- Never fabricate information\n- Pricing questions -> /pricing page\n\"\"\"\n\n# api_handler.py\nfrom prompts import SUPPORT_SYSTEM_PROMPT\n\n# worker.py\nfrom prompts import SUPPORT_SYSTEM_PROMPT\n```\n\n\n### 3. Few-Shot Example Rotation\n\nAs mentioned earlier, shuffling or rotating few-shot examples per request breaks prefix matching. But there is a subtler version of this: adding new examples to the set. If you append a new example to your few-shot block, every request after the addition gets a cache miss on the new prefix until it warms up. Plan your example updates in batches during low-traffic periods.\n\n### 4. Not Monitoring Cache Hit Rates\n\nThis is the meta-pitfall. Everything else on this list is a specific mistake, but not monitoring means you will never catch any of them. A code change six months from now might introduce a dynamic element into your prefix. Without monitoring, you will only notice when your monthly bill goes up.\n\nBuild the tracker from the previous section into your production logging. Alert on hit rate drops below your threshold (80% is a reasonable starting point). Treat cache hit rate as a first-class production metric, just like latency and error rate.\n\n---\n\n## Exercise\n\n**Build a cache-optimized chatbot and measure the savings.**\n\nTake a multi-turn chatbot from a previous module (or build a simple one) with a system prompt of at least 1,500 tokens. Include role instructions, behavioral guidelines, and at least 5 few-shot examples.\n\n1. **Restructure for caching.** Move all static content to the front, all dynamic content to the end. For Anthropic, add `cache_control` breakpoints. For OpenAI, ensure the prefix exceeds 1,024 tokens and is byte-identical across calls.\n\n1. **Run 100 requests** with varied user messages but the same system prompt and few-shot examples. Use the `CacheStats` tracker from this chapter to log every request.\n\n1. **Measure results** on both Anthropic and OpenAI:\n\n- What is your cache hit rate?\n- What is your token cache rate?\n- How much did you save in dollar terms compared to no caching?\n\n1. **Intentionally break the cache** by introducing a timestamp into the system prompt. Run another 100 requests and compare the hit rate and cost to the optimized version.\n\n1. **Write a short report** documenting the before/after numbers and what you learned about prefix design.\n\n**Target:** Cache hit rate above 95% on the optimized version, below 5% on the broken version. Cost savings of at least 40% on Anthropic and 20% on OpenAI.\n\n---\n\n## Summary\n\nHere are the key takeaways from this chapter:\n\n- **Prompt caching stores precomputed KV matrices for repeated prompt prefixes.** This lets the model skip the expensive attention computation for tokens it has already processed, saving both cost and latency.\n\n- **Anthropic offers explicit caching with the deepest discount.** You mark blocks with `cache_control`, pay a 25% write surcharge on the first request, and get 90% off on all subsequent cache hits. The 5-minute TTL resets on each hit, so steady traffic keeps the cache warm.\n\n- **OpenAI caching is automatic and free to set up.** No code changes needed. Prefixes over 1,024 tokens get cached transparently with a 50% discount. Just keep your prefixes consistent.\n\n- **Google Gemini is designed for large contexts.** Create explicit cache objects for documents over 32K tokens, reference them across requests, and get 75% off. Best for document QA, codebase analysis, and similar large-context workflows.\n\n- **Prompt structure determines cache hit rate.** The golden rule is static content first, dynamic content last. System instructions, tool definitions, and few-shot examples belong at the top. User messages, timestamps, and retrieved context belong at the bottom.\n\n- **Monitor cache performance as a production metric.** Track hit rate, token cache rate, and cost savings. Alert on drops. A single code change can silently destroy your cache hit rate.\n\nPrompt caching works at the API level, but what exactly is being cached? Those \"KV matrices\" we keep mentioning are the key-value pairs computed at each attention layer inside the transformer. In the next chapter, we will look inside the model to understand the KV cache: how it grows during generation, why it dominates GPU memory, and what techniques like multi-query attention and paged attention do to keep it under control.\n\n---\n\n## References\n\n- [Anthropic Prompt Caching Documentation](undefined)\n- [OpenAI Prompt Caching Guide](undefined)\n- [Google Gemini Context Caching Documentation](undefined)\n- [Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023)](undefined)\n- [Efficiently Scaling Transformer Inference (Pope et al., 2022)](undefined)","pageType":"ai-engineering"}

11.6 Prompt Caching

Ashish Pratap Singh

Get Premium