> Source: Commit 515b759, project easy-learn-ai, published 2026-05-06
When AI Learns Perfect Recall: How Prompt Caching Saves You Money
Have you ever wondered what actually happens on the backend every time you chat with ChatGPT or Claude?
Not the hand-wavy "the AI is thinking"—but concrete engineering details. Every character you send gets re-processed by the server. System instructions, tool definitions, chat history, your brand-new message—all merged together and computed from the very first token to the last. This is called prefill, and it's the biggest chunk of both latency and your bill.
If you've chatted for 20 turns, the first 19 turns in turn 20's request are exactly identical to last time. The model still has to recompute from scratch. It's like rewriting the table of contents and the first chapters of your thesis every time you sit down to add a new paragraph.
That sounds dumb, right? It is. So someone came up with a fix: store the computation you've already done. That's Prompt Cache.
---
The Principle in One Sentence
You mark a breakpoint in your request. The backend stores the "encoding result" from the start up to that breakpoint. Next time, if the prefix is exactly the same, it's reused directly—no repeated computation.
It's like copying your thesis up to chapter three and putting a bookmark there. Next time, fast-forward through the first three chapters and only write the new content.
---
How Much Do You Save? Anthropic Gives Real Numbers
Key figures to remember:
- Cache hits cost 10% of the normal price (90% savings)
- First write costs 1.25x—25% more, but every subsequent read saves 90%
- Default TTL is 5 minutes; any request within 5 minutes renews it for free; a paid 1-hour option exists
- Minimum 1024 tokens to be cacheable (some newer models require 4096); short prompts don't qualify
- System instructions: fixed, inside the cacheable prefix
- System messages: fluid, don't affect the prefix
A concrete example: a 100k-token long conversation. Without caching, Claude Sonnet costs $0.30 per turn. With caching, the first turn costs $0.375, then $0.03 per turn. Over 10 turns, you save roughly 90% of input costs.
And it's not just money. Latency drops too—the more you skip recomputing, the faster the first token appears. That metric is TTFT (Time To First Token), and it hugely affects interactive experience.
---
Anthropic Monitors Cache Hit Rate Like Server Uptime
This surprised me the most.
Internally at Anthropic, cache hit rate is an infrastructure-level metric with the same standing as server availability. If hit rate drops, on-call alerts fire and engineers treat it as a live incident—the original text uses the phrase "declare SEVs," meaning a full incident-response process.
Why so serious? Because without caching, there is no Claude Code.
An AI coding assistant like Claude Code runs sessions of dozens of turns. Every turn resends the full context. Recomputing from scratch each time would explode latency and cost. Caching isn't a nice-to-have optimization—it's the precondition for the whole system to work.
Even more importantly, a high hit rate directly improves user experience—it lets Anthropic offer paid users more generous usage limits. The higher the hit rate, the more you can use at the same price.
---
Core Principle: Prefix Matching
The core logic is four words: prefix matching. If the next request's prefix matches the previous one, the prior computation can be reused.
It looks simple, but the constraint is brutal—a change at any position in the prefix invalidates the cache for everything after it. Like dominoes: knock over the first one and everything behind falls.
So all the "best practices" below are really doing the same thing: keep as much of the prefix unchanged as possible.
---
Line Things Up: The Most Stable Content Goes First
Anthropic's best practices specify an ordering for prompt content:
1. Front: system instructions + tool definitions (shared across all sessions, fixed) 2. Second layer: project documentation (shared within a project, relatively stable) 3. Third layer: current session context (valid only for this conversation) 4. Last: chat messages (grow turn by turn; each turn only appends)
In one line—the least changeable content goes first. Like organizing a desk: reference books you rarely touch go on the bottom shelf, this week's reading in the middle, today's draft on top. That way you never have to reshuffle the whole desk each time you sit down.
---
Three Especially Easy Pitfalls
Pitfall 1: Embedding the Current Time in Fixed Instructions
Your system prompt says "today is May 6, 2026." The next second it's May 7 and the cache is dead. Put time information into chat messages, never touch the system prompt.
Pitfall 2: Storing Tool Definitions in Unordered Containers
HashMaps, Sets—these can iterate in different orders each time. The tool definitions' order changes, the prefix no longer matches, cache invalidated. Always use ordered lists.
Pitfall 3: Changing Even One Field of a Tool Parameter
Adding a parameter, changing a type, deleting a field—the entire prefix cache is invalidated. The tool definitions changed, so the prefix is no longer the same.
---
Advanced Technique 1: Plan Mode
Claude Code has a "planning mode." When active, the model only thinks and plans—it doesn't execute actions.
The intuitive approach: enter planning mode by removing execution tools, and add them back when exiting. But Anthropic doesn't do that. They keep all tools in place and add two special tools—"enter planning" and "exit planning."
The model calls "enter planning" to switch into thinking mode, and "exit planning" to come back. How is the "no execution in planning mode" constraint communicated? By inserting a system message into the conversation, telling the model it's now planning.
Note the distinction—insert a message into the conversation flow, don't modify the system instructions:
---
Advanced Technique 2: Lazy Loading
Claude Code may integrate dozens of external tools. Stuffing in full definitions for all of them? Too much space. Adding/removing on demand? Breaks the cache.
Anthropic's compromise is lazy loading. At first, only a lightweight placeholder is included—the model sees just the tool name, without full parameter definitions. When the model actually needs a tool, it fetches the full definition via a "tool search" feature.
The benefit: the prefix only ever contains those lightweight placeholders, so it doesn't change when a tool gets loaded. Cache stays stable.
It's like a library catalog: you browse the index first, then fetch the book from the shelf—no need to pile every book on your desk.
---
Advanced Technique 3: Cache-Safe Forking
Long conversations eventually fill the context window. At that point you compress prior conversation into a summary to free up space.
But here's the problem. If you spin up a separate request for compression with different system instructions and no tool definitions—from the very first token it won't match the main conversation's cache. Two cache chains, no mutual reuse. Plus you have to send the whole conversation for summarization, paying full, non-discounted prices. The longer the conversation, the more expensive.
Anthropic's solution is "cache-safe forking." The compression request must use exactly the same system instructions, user context, and tool definitions as the main conversation, carrying the main conversation's messages as history. Then append one compression instruction at the end, as a new user message.
From the backend's perspective, this request is nearly identical to the last one. Same prefix, same tools, same history—so the prefix cache is reused directly. The only new cost is the compression instruction itself.
You also need to reserve a compaction buffer, leaving enough room for the summary output. Don't wait until the window is completely full—leave slack in advance.
---
Sub-Agents: Don't Pull the Intern to Your Desk
Sometimes you need a smaller model to do the work—like Claude Code's exploration mode, where a small model performs search tasks in an independent context.
The correct approach: the subtask has its own independent context and cache, so it doesn't pollute the main conversation's cache chain. Have the main model write a task handoff note condensing the context, pass it to the subtask for execution, and return only the results to the main conversation.
Analogy: you wouldn't have an intern sit at your desk using your computer. You'd assign them a separate machine, write clear instructions, send them over, and get results back.
A warning for people running proxy/relay services—caches are isolated per account. Some try using account pools for relaying; once accounts get mixed, hit rates crash, and you lose money before you lose the accounts.
---
Looking Back: All the Lessons Say the Same Thing
Summarizing the lessons:
1. Prefix matching decides everything. A change at any position in the prefix invalidates the cache for everything after it. 2. Use messages instead of instruction modifications. To switch modes or update time, put it in the conversation—don't touch the system prompt. 3. Don't switch tools or models mid-conversation. Express state transitions with tools; use lazy loading instead of adding/removing tools. 4. Monitor cache hit rate like uptime. Anthropic alerts on cache degradation and treats it as a live incident. 5. Forking operations must share the main conversation's prefix. Compression, summarization, subtasks—all use identical parameters.
Every design decision should revolve around this one constraint. Don't change instructions, don't touch tools—one touch breaks the whole cache chain. Switching models, switching accounts, starting fresh—same principle.
---
Finally
This looks like cache optimization, but it's also a mindset—first accept the one constraint you can't escape, then build the whole system around it.
The Prompt Cache story is, at its core, a triumph of engineering intuition. While everyone debates bigger models and more parameters, someone looked back at the bill and said: wait, are we repeating the same computation over and over?
The answer was yes. And the fix was a simple "prefix matching."
#easy-learn-ai #daily-update #memory