When AI Learns to "Take Notes": How Prompt Caching Turns LLMs from Re-Copying Everything into Fast-Forwarding

Source commit: 515b759

Have you ever wondered what actually happens behind the scenes every time you send a message to ChatGPT or Claude?

It's not just the model "thinking." In reality, every new message you send forces the server to re-encode everything you've said before — system instructions, tool definitions, conversation history, plus your new message — from scratch. This process is called prefill, and it's the biggest driver of both latency and cost.

Here's the absurd part: after 20 turns of conversation, turn 20's request contains 19 turns of content identical to last time — yet the model still recomputes from the very first token. It's like having to re-copy the table of contents and earlier chapters of your thesis every time you add a new paragraph. Nineteen rounds of wasted work.

Prompt caching exists to fix exactly this.

1. How it works: prefix matching, like fast-forwarding when copying a book

The mechanism in one sentence: you mark a breakpoint in the request, and the backend stores the encoding results from the start up to that breakpoint. Next time, if the prefix is exactly identical, it reuses the stored results and skips the repeated computation.

Analogy: without caching, you re-copy the table of contents and earlier chapters each time before adding new content. With caching, the already-copied parts fast-forward, and you only write the new section.

This is a technique Anthropic uses extensively in Claude Code, its AI coding assistant. A single session can run dozens of turns, each carrying the full prior context. Without caching, latency and cost would explode.

Internally, Anthropic treats cache hit rate as an infrastructure-level metric — on par with server uptime. If the hit rate drops, it triggers on-call alerts and engineers treat it as a production incident. The original wording was "declaring a分级 (sev-level) incident," which is far more serious than a normal alert.

Crucially, a high hit rate doesn't just save money — it directly improves user experience, letting Anthropic offer paid users more generous usage limits. The higher the hit rate, the more you can use at the same price.

For Claude Code, caching isn't a nice-to-have optimization. It's the precondition for the system to work at all. No cache, no Claude Code.

2. The numbers: a 90% discount

Key figures:

• Cached input costs 10% of the base input price
• First cache write costs 1.25x — a 25% premium
• Every subsequent hit saves 90%

The default cache TTL is 5 minutes. Any request within that window renews it for free. A paid 1-hour option also exists.

There's a threshold — content that's too short can't be cached. Generally at least 1,024 tokens; newer models require 4,096. Short prompts get no cache.

A concrete example: a 100k-character long conversation with Claude Sonnet.

• Without caching: $0.30 per turn
• With caching: $0.375 the first time, then only $0.03 per turn

Over 10 turns, that's roughly 90% saved on input costs. And it's not just money — latency drops too. The more that doesn't need recomputation, the faster the first token appears.

3. Order matters: the more stable, the earlier

Since caching relies on prefix matching, the arrangement of prompt components is critical.

Anthropic's best-practice ordering: 1. Front: system instructions and tool definitions. Fixed, shared across all sessions. 2. Second layer: project documentation (CLAUDE.md). Shared within the same project. 3. Third layer: current session context. Valid only for this conversation. 4. Last: chat messages. Grows turn by turn; each turn only appends one message.

One-line summary: the less likely something is to change, the earlier it goes.

It's like organizing a desk: reference books you rarely touch go on the bottom shelf, this week's materials in the middle, today's drafts on top. That way you don't have to翻 the whole desk every time you sit down.

4. Three pitfalls: one small detail breaks the whole cache chain

• Pitfall 1: Embedding the current time in fixed instructions. It changes every second — cache instantly dead.
• Pitfall 2: Tool definitions in unordered containers, like Python dict or set. The order differs on each request, so prefixes never match.
• Pitfall 3: Changing tool parameters. Even one modified field invalidates the entire prefix cache.

One overlooked detail, and the whole cache chain is broken.

5. Need to update information? Send a message, don't touch the prompt

During task execution, information goes stale — built-in timestamps drift, users edit files. The obvious instinct is to update the system prompt, but that immediately invalidates the cache, which is costly for users.

Claude Code's approach is to deliver these updates as the next conversational message instead of modifying the prompt prefix. They quietly inject a <system-reminder> tag into the next user message or tool result to tell the model the latest information.

This small trick perfectly preserves the precious cache.

6. Don't switch models, don't touch tools

You might think: switch to Haiku for easy questions, back to Opus for hard ones — smart, right?

In reality, cache is bound to the model. Switching models voids all accumulated cache, forcing a full rebuild — often more expensive than just letting Opus answer the easy question.

So Claude Code's strategy: keep the same model for the entire main conversation.

What if you need a smaller model to do work? Dispatch a sub-task. Sub-tasks have their own independent context and cache, so they don't pollute the main conversation's cache chain. Concretely: the main model writes a task handoff summarizing the context, passes it to the sub-task, and only the result comes back.

Analogy: you don't let an intern sit at your desk using your computer. You assign them a separate machine, write clear instructions, and have them send results back.

A warning for anyone running proxy/relay services: cache is isolated per account. People mixing account pools end up with abysmal hit rates — they lose money and get accounts banned. Also be wary of advice telling you to rapidly switch accounts. Don't flip-flop every couple of messages.

7. Compress context, but stay cache-safe

Long conversations eventually fill the context window, so you compress past dialogue into a summary to free space.

The problem: if you issue a separate API call for compression with a different system prompt and no tool definitions, it mismatches the main conversation's cache from the very first token. Two cache chains, zero reuse — paying twice for nothing.

Anthropic's solution is Cache-Safe Forking:

The compression request must use exactly the same system instructions, user context, and tool definitions as the main conversation, carrying the main conversation's messages as history. Then append the compression instruction as a new user message at the end.

From the backend's view, this request is nearly identical to the previous one — same prefix, same tools, same history — so the prefix cache is directly reusable. The only new cost is the compression instruction itself.

You should also reserve a compression buffer so the summary has room to output. Don't wait until the window is completely full — leave headroom in advance.

One compression operation, reusing all the accumulated cache. Almost no extra cost.

8. Plan mode and lazy loading

Claude Code has another clever design called Plan mode.

When a user requests something complex, the model doesn't act immediately. It enters a "planning state" — analyzing the task, breaking down steps, listing tools to call. This state transition is not done by modifying the system prompt (which would break the cache), but by simulating it through a tool call.

The model calls a special plan tool to signal "entering planning mode." The system recognizes the call and displays the planning UI. Throughout, the system prompt doesn't change by a single character — cache intact.

Similarly, lazy loading is cache-friendly: instead of loading all tool definitions upfront, tools load on demand. This keeps the static prefix minimal and the cache hit rate higher.

9. Takeaways for you

Whether you use Claude Code or build your own agent from scratch, these rules apply: 1. Order your prompts: system instructions → tool definitions → reference docs → conversation history 2. Update information via messages, never by editing the system prompt 3. Never switch models or tools mid-conversation 4. Use sub-agents for tasks requiring different models 5. Use cache-safe forking when compressing context 6. Monitor cache hit rate like you monitor uptime

Claude Code was designed around prompt caching from day one. If you're building the next great agent, start from this rule too.

References:

• Anthropic blog: "Lessons from building Claude Code: Prompt caching is everything"
• easy-learn-ai project: interactive prompt-cache presentation site