Cache Rules Everything: How AI Now "Remembers" Every Word You've Said
> Source: easy-learn-ai commit 515b759
> Inspired by: Anthropic engineers' sharing of best practices for Claude Code prompt caching design
1. The Problem: Why Does the AI Get "Slower" the Longer You Chat?
Imagine chatting with a brilliant friend all afternoon, over twenty turns. After each thing you say, he falls silent in thought — and then you realize he isn't thinking about your latest sentence. He is re-reading everything you've discussed, from the very first sentence, word for word, in his head.
You'd think he'd gone mad.
But this is how almost every large language model works today.
Every time you send a new message, the backend must re-encode all the text — system instructions, conversation history, tool definitions, the new message — from the beginning. This process is called Prefill, and it's the main driver of both latency and cost.
After 20 turns, the first 19 turns of your 20th request are identical to last time. But the model still recomputes from the first token, doing 19 turns of duplicated work for nothing.
This isn't an optimization problem. It's structural waste.
---
2. Prompt Caching: Giving the AI a "Notepad"
Prompt caching solves this.
The principle in one sentence: you mark a breakpoint in the request. The backend stores the encoding of everything up to that point. Next time the prefix is identical, it's reused directly, skipping the repeated computation.
It's like writing a paper: without a cache you'd re-copy the table of contents and earlier chapters before adding new content. With a cache, the copied parts fast-forward and you only write the new section.
But this isn't a minor trick to save a little time. Let's look at the real numbers.
---
3. The Numbers Don't Lie: 90% of Cost Can Be Saved
Anthropic publishes a few key ratios:
- Cache hits cost one tenth of the price (cache reads = 10% of the normal rate)
- First write costs 25% extra (cache writes = 125%)
- Every subsequent request saves 90%
There is a threshold — content that's too short can't be cached. Generally at least 1,024 tokens, and 4,096 for newer models. Short prompts get no cache.
A concrete example: a 100k-character long conversation.
Without caching, each turn on Claude Sonnet costs $0.30. With caching, the first turn costs $0.375 (25% extra), and each turn after that only $0.03. Over 10 turns, you save roughly 90% of input cost.
And it's not just money. Latency drops too — the more that doesn't need recomputation, the faster the first token arrives (TTFT, Time To First Token).
There's more. Anthropic treats cache hit rate as an infrastructure-level metric, on par with server uptime. When the hit rate drops, it triggers on-call alerts, and engineers handle it like a production incident — the original text uses the phrase "declare SEVs."
More importantly, a high hit rate doesn't just save money; it directly improves user experience — it lets Anthropic offer paid users more generous usage limits. The higher the cache hit rate, the more you can use at the same price.
So caching isn't a nice-to-have optimization for Claude Code. It's the precondition for the whole system to work. No cache, no Claude Code.
---
4. The Core Principle: Prefix Matching
To understand prompt caching, remember four words: prefix matching.
If the next request's prefix matches the last one, the previous computation is reused. But that means — any change anywhere inside the prefix invalidates the cache for everything after it.
Like a chain: change any link in the middle, and every link after it must be redone.
Since caching relies on prefix matching, the ordering of components in the prompt is critical. Anthropic's best practice ordering:
| Layer | Content | Shared across | |-------|---------|---------------| | 1 | System instructions + tool definitions | All sessions | | 2 | Project docs (CLAUDE.md) | Same project | | 3 | Current session context | This conversation only | | 4 | Chat messages | Grows each turn; only the last message is new |
One sentence: put the things that change least at the front.
It's like organizing a desk: rarely-touched reference books at the bottom, this week's materials in the middle, today's draft on top — so you don't have to redo the whole desk every time you sit down.
---
5. Seven Pitfalls: Lessons Anthropic Already Learned For You
Pitfall 1: Embedding the current time in fixed instructions
It changes every second, instantly killing the cache.
Fix: Don't modify the system instructions; push updates into the next message. Attach a "system reminder" snippet inside the next user message. System instructions are the foundation — nail them down; messages are flowing water — change them freely.
Pitfall 2: Storing tool definitions in unordered containers
If the order differs between requests, the prefix won't match.
Fix: Tool definitions must keep a fixed, deterministic order.
Pitfall 3: Changing tool parameters
Even a single changed field invalidates the entire prefix cache.
Counterintuitive trap: The current task only needs 3 tools — wouldn't removing the other 30 be cleaner?
The truth: Tool definitions are part of the cached prefix. Adding or removing one breaks the cache. Once broken, the whole conversation's cache must be rebuilt — far more costly than the space a few extra definitions take. What looks like optimization turns into sabotage.
Pitfall 4: Switching models
You might think: route easy questions to a small model to save money, hard ones back to the big model. Sounds sensible.
But in reality, caches are model-bound. Switching models voids all accumulated cache and forces a full rebuild — often more expensive than just letting the big model answer the easy question.
So Claude Code's policy is — the main conversation uses one model from start to finish. Need a small model for something? Dispatch a subtask. Subtasks have their own independent context and cache and don't pollute the main conversation's cache chain.
The concrete approach: have the main model write a task handoff brief condensing the context, pass it to the subtask, and only pass the result back. You wouldn't let an intern sit at your desk using your computer. You'd assign them a separate machine, send clear instructions, and get the results back.
Account warning: Caches are isolated per account. Some people try account pools for relaying. Once accounts get mixed, the hit rate collapses — no money saved and accounts get banned.
Pitfall 5: Tool switching for plan mode
Claude Code has a "plan mode" where the model only thinks and plans, without executing actions.
The intuitive approach: remove execution tools on entering plan mode, add them back on exit.
That's not what Anthropic does. They keep all tools in place, and instead add two special tools — "enter planning" and "exit planning." The model calls "enter planning" to switch into thinking mode, "exit planning" to come back.
How is the "no execution in plan mode" constraint conveyed? By inserting a system message into the conversation telling the model it's now planning.
Note — the message is inserted into the conversation flow, not by editing the system instructions. Keep these distinct. System instructions are fixed and live in the cached prefix. Conversation messages flow and don't affect the prefix. The tool set never changes, so the cache stays valid.
A bonus: the model can decide for itself when to plan — for complex tasks it thinks first without you manually switching.
Pitfall 6: Stuffing in dozens of full tool definitions
Claude Code may integrate dozens of external tools. Full definitions for all? Too much space. Add/remove on demand? Breaks the cache.
Anthropic's compromise is lazy loading.
At the start, only lightweight placeholders are included — the model sees tool names without full parameter definitions. When the model actually needs a tool, it fetches the full definition via a "tool search" feature.
Benefit: the prefix always contains only the lightweight placeholders, never changing because a tool got loaded. The cache stays solid.
It's like a library's card catalog: browse the index first, then fetch the book — no need to drag every book onto your desk.
Worth noting: this tool search capability has been exposed via API, so developers can use it to simplify their own tool management.
Pitfall 7: Context compression starting from scratch
Long conversations fill up the context window. The fix: compress the prior dialogue into a summary to free space.
The problem: if you spin up a separate request for compression — with different system instructions and no tool definitions — nothing matches the main conversation's cache from the very first token. Two cache chains, no reuse between them.
Worse, you must send the entire conversation for summarization, paying full, undiscounted prices. The longer the conversation, the more expensive.
Anthropic's solution is "cache-safe forking."
The compression request must use exactly the same system instructions, user context, and tool definitions as the main conversation, carrying the main conversation's messages as history. Then append one compression instruction as a new user message at the end.
From the backend's view, this request is nearly identical to the previous one. Same prefix, same tools, same history — so the prefix cache is reused directly. The only new cost is the compression instruction itself.
You also need a compression buffer reserved for the summary output. Don't wait until the window is completely full — leave headroom in advance.
One compression operation reuses all the cache the main conversation accumulated. Almost no extra cost.
---
6. Looking Back: Every Lesson Says the Same Thing
Looking at these 7 lessons, they all say one thing — cache is prefix matching.
1. Prefix matching decides everything. Any change anywhere in the prefix invalidates everything after it. 2. Use messages instead of editing instructions. To switch modes or update time-sensitive info, put it in conversation messages; don't touch system instructions. 3. Don't switch tools or models mid-conversation. Express state transitions via tools; replace tool add/remove with lazy loading. 4. Monitor cache hit rate like uptime. Anthropic alerts on cache disruptions and treats them as production incidents. 5. Fork operations must share the main conversation's prefix. Compression, summarization, subtasks — all use identical parameters.
All design should revolve around this constraint. Don't touch the instructions, don't touch the tools — one touch, and the whole cache chain breaks. Switching models, switching accounts, starting from scratch — same principle.
This looks like cache optimization, but it also teaches a mindset — first accept the unbreakable constraint, then build the whole system around it.
---
7. Why This Matters
Prompt caching is not a fringe optimization. It's infrastructure for the AI agent era.
Without it, every AI coding assistant, every multi-turn dialogue system, every tool-call-heavy agent would suffocate under the dual pressure of cost and latency. A 100k-character conversation at $0.30 per turn costs $30 over 100 turns — not counting output tokens.
With it, the same conversation costs a tenth as much, with response speeds users can't perceive waiting for.
Engineers have an old saying: "cache rules everything." In the AI agent era it still holds — except this time the cache doesn't store web pages or images. It stores the state of an AI's half-finished thinking.
---
> 📌 This article's content comes from the easy-learn-ai project's interactive tutorial "Prompt Cache, Made Easy" — an immersive learning experience with 15 chapters, audio narration, and stage animations. If you're interested in the technical details, go experience it yourself.