Source: Commit 515b759, easy-learn-ai / "Understand Prompt Cache Easily", published 2026-05-06.
When AI Learns Perfect Recall: How Prompt Cache Saves You Money
Have you ever wondered what actually happens behind the scenes every time you chat with ChatGPT or Claude?
Not the hand-wavy "AI is thinking" stuff, but the concrete engineering details — every character you send gets re-processed by the server. System instructions, tool definitions, chat history, and your brand-new message all get mixed together and computed from the very first token. This process is called prefill, and it's the biggest chunk of both latency and your bill.
So after 20 turns of conversation, the 20th request contains the previous 19 turns exactly identical to last time. The model still computes from scratch. It's like having to re-copy the table of contents and the first few chapters every time you continue writing a paper.
That sounds dumb, right? It is. So someone came up with a solution: store the already-computed parts. That's Prompt Cache.
---
The Principle in One Sentence
You mark a breakpoint in your request. The backend stores the "encoded result" from the start up to that breakpoint. Next time, if the prefix is exactly the same, it's reused directly, skipping the redundant computation.
It's like bookmarking chapter three of your paper. Next session, fast-forward through the first three chapters and only write the new part.
---
How Much Does It Save? Anthropic's Real Numbers
A few key figures to remember:
- Cache hits are priced at one-tenth (90% savings)
- First write costs 1.25x — 25% extra, but every subsequent use saves 90%
- Default 5-minute TTL, automatically refreshed by requests within the window at no extra cost; a paid 1-hour option is also available
- Minimum 1024 tokens to qualify (some newer models require 4096); short prompts don't get caching
- System instructions: fixed, part of the cache prefix
- System messages: flowing, don't affect the prefix
A concrete example: a 100k-token long conversation. Without caching, each turn costs $0.30 on Claude Sonnet. With caching, the first turn costs $0.375, then $0.03 per turn after. Over 10 turns, that's roughly 90% saved on input costs.
And it's not just money. Latency drops too — the more you skip recomputing, the faster the first token arrives. This metric is TTFT (Time To First Token), and it hugely affects interactive experience.
---
Anthropic Monitors Cache Hit Rate Like Server Uptime
This is the most surprising part.
Internally at Anthropic, cache hit rate is an infrastructure-level metric with the same standing as server uptime. When the hit rate drops, it triggers on-call alerts, and engineers treat it as a live incident — the phrase used is "declare SEVs," meaning the full incident-response process.
Why take it so seriously? Because without caching, there is no Claude Code.
An AI coding assistant like Claude Code runs sessions of dozens of turns. Each turn re-sends the full context. Recomputing from scratch every time would make latency and cost explode. Caching isn't a nice-to-have optimization — it's the precondition for the whole system to work.
More importantly, a high hit rate doesn't just save money — it directly improves user experience, letting Anthropic offer paid users more generous usage limits. The higher the cache hit rate, the more you can use at the same price.
---
The Core Mechanism: Prefix Matching
The core logic is four words: prefix matching. If the next request's prefix matches the previous one, the previous computation is reused.
Sounds simple, but the constraint is brutal — a change at any position in the prefix invalidates the cache for everything after it. Like dominoes: one block falls, everything behind it starts over.
So all the "best practices" that follow are essentially doing the same thing: keep as much of the prefix unchanged as possible.
---
Line Things Up: The More Stable, The Earlier
Anthropic's best practices prescribe a specific prompt ordering:
1. Front: system instructions + tool definitions (shared across all sessions, fixed) 2. Second layer: project documentation (shared within a project, relatively stable) 3. Third layer: current session context (valid only for this conversation) 4. Last: chat messages (grows each turn, only the last one is new)
In one line — put the least-changeable things first. Like organizing a desk: rarely-touched reference books on the bottom shelf, this week's reading in the middle, today's draft on top. That way you don't have to flip the whole desk every time you sit down.
---
Three Easy Traps to Fall Into
Trap 1: Embedding the current time in fixed instructions
System prompt says "today is May 6, 2026." A second later it's May 7, and the cache is dead. Put time into the conversation messages instead — never touch the system instructions.
Trap 2: Storing tool definitions in unordered containers
HashMaps, Sets, and the like may iterate in different orders each time. The tool definitions arrive in a different order, the prefix doesn't match, cache invalidated. Always use ordered lists.
Trap 3: Changing even one field of a tool's parameters
Add a parameter, change a type, delete a field — the entire prefix cache is invalidated. Tool definitions changed, so the prefix is no longer the same.
---
Advanced Technique 1: Plan Mode
Claude Code has a "plan mode" where the model only thinks and plans, without executing actions.
The intuitive approach: enter plan mode by removing execution tools, then add them back on exit. But Anthropic doesn't do that. They keep all tools in place and add two special tools — "enter planning" and "exit planning."
Calling "enter planning" switches the model to thinking mode; "exit planning" returns it. How is the constraint "no execution while planning" conveyed? By inserting a system message into the conversation, telling the model it's now in planning mode.
Note — a message inserted into the conversation flow, not a change to the system instructions. These are two different things:
---
Advanced Technique 2: Lazy Loading
Claude Code may integrate dozens of external tools. Fully defining all of them? Too much space. Adding and removing on demand? Breaks the cache.
Anthropic's compromise is lazy loading. Initially only lightweight placeholders are included — the model sees tool names without full parameter definitions. When it actually needs a tool, it fetches the full definition via a "tool search" feature.
The benefit: the prefix always contains only lightweight placeholders, never changing just because a tool got loaded. Cache stays rock-solid.
It's like a library's catalog index. Browse the catalog first, then fetch the book you want — no need to move the entire library onto your desk.
---
Advanced Technique 3: Cache-Safe Forking
Long conversations eventually fill the context window. At that point you need to compress the history into a summary to free up space.
Here's the problem. If you fire off a separate request for compression with different system instructions and no tool definitions, it diverges from the main conversation's cache from the very first token — two cache chains, no mutual reuse. And since you must send the whole conversation for summarization, you pay full, undiscounted prices. The longer the conversation, the more expensive.
Anthropic's solution is called "cache-safe forking." The compression request must use exactly the same system instructions, user context, and tool definitions as the main conversation, carrying the main conversation's messages as history. Then append one compression instruction at the end as a new user message.
From the backend's perspective, this request looks almost identical to the previous one. Same prefix, same tools, same history — so the prefix cache is directly reusable. The only new cost is the compression instruction itself.
You should also reserve a compaction buffer, leaving enough room for the summary output. Don't wait until the window is full — keep margin ahead of time.
---
Sub-Agents: Don't Pull the Intern Onto Your Desk
Sometimes you need a smaller model to do work — e.g., Claude Code's exploration mode uses a small model to run search tasks in an independent context.
The right approach: subtasks have their own independent context and cache, never polluting the main conversation's cache chain. The main model writes a task handoff brief, condensing the context. The subtask executes it and sends back only the results.
An analogy: you wouldn't have an intern sit at your desk using your computer. You'd assign them a separate machine, send a clearly written task brief, and get results back.
A warning for those running proxy/relay setups — caches are isolated per account. Some people try account pools for relaying; once accounts get mixed, hit rates tank, and you lose money before you lose the accounts.
---
Looking Back: Every Lesson Says the Same Thing
The seven lessons, summarized:
1. Prefix matching decides everything. A change at any position in the prefix invalidates everything after it. 2. Use messages instead of instruction edits. To switch modes or update the time, put it in conversation messages — never touch system instructions. 3. Don't switch tools or models mid-conversation. Express state transitions via tools; use lazy loading instead of adding/removing tools. 4. Monitor cache hit rate like uptime. Anthropic fires alerts on cache disruptions and treats them as production incidents. 5. Forked operations must share the main conversation's prefix. Compression, summarization, subtasks — all use identical parameters.
Every design decision revolves around this one constraint. Don't change instructions, don't touch tools — one touch, and the whole cache chain breaks. Switching models, accounts, or starting fresh — same principle.
---
Finally
This looks like cache optimization, but it's really a mindset — first accept the one non-negotiable constraint, then build the entire system around it.
The Prompt Cache story is, at its heart, a triumph of engineering intuition. While everyone was debating bigger models and more parameters, someone looked at the bill and said: wait, aren't we doing the same work over and over?
The answer was yes. And the fix was a simple "prefix matching."
Tags from source: easy-learn-ai, daily updates, memory, XiaoKai.