How Prompt Caching Lets AI Stop Recomputing Everything: Lessons from Claude Code

Forum topic · 小凯 · 2026-05-09

Summary

This post explains prompt caching in large language models, based on Anthropic's engineering insights into Claude Code. Every new message normally requires re-encoding the entire context—system instructions, tool definitions, and conversation history—making prefill the dominant cost and latency source. Prompt caching stores the encoding of a marked prefix and reuses it when the prefix matches exactly, cutting cached input costs to 10% of base price (with a 1.25x first-write fee). The article covers Anthropic's ordering best practices (stable content first: system prompt, tools, project docs, then messages), common cache-breaking pitfalls (embedding timestamps, unordered containers for tool definitions, mid-conversation model switching), and advanced techniques: passing updates as messages instead of editing system prompts, cache-safe forking for context compression, plan mode via tool calls, lazy tool loading, and sub-agents for different models. Cache hit rate is treated as critical infrastructure at Anthropic, since high hit rates enable cheaper, faster responses and looser usage quotas.

Source commit: 515b759

Have you ever wondered what actually happens in the background each time you send a message to ChatGPT or Claude?

It's not simply "thinking." Every new message forces the server to re-encode everything that came before it—system instructions, tool definitions, conversation history, plus your new message—from scratch. This process is called prefill, and it's the biggest driver of both latency and cost.

Worse: after 20 turns of conversation, turns 1–19 are identical to the previous request, yet the model still recomputes from the first token. It's like copying the table of contents and first chapters of a paper every time you add a new paragraph.

Prompt caching solves exactly this.

1. How It Works: Prefix Matching, Like Skipping Ahead

The mechanism in one sentence: you mark a breakpoint in the request, and the backend stores the encoding result from the start up to that breakpoint. Next time the prefix matches exactly, it's reused and duplicate computation is skipped.

This is the technology Anthropic uses extensively in Claude Code, its AI coding assistant where a single session can span dozens of turns—each carrying the full prior context.

Internally, Anthropic treats cache hit rate as an infrastructure-level metric, on par with server uptime. A drop in hit rate triggers on-call alerts and is handled as a production incident (what the source calls "declaring a Sev-level incident").

Crucially, high hit rates don't just save money—they let Anthropic offer paying users more generous usage quotas. For Claude Code, caching isn't a nice-to-have optimization. It's a precondition for the system working at all.

2. The Numbers: A 90% Discount

Key figures:

Cached portions cost 10% of base input price (a 90% discount)
First write costs 1.25x—a 25% premium
Every subsequent use saves 90%

The default cache TTL is 5 minutes, auto-renewed on each request at no extra charge. A paid 1-hour option exists.

There's a threshold: content that's too short can't be cached—typically at least 1,024 tokens (4,096 for newer models).

A concrete example: a 100,000-character conversation with Claude Sonnet:

Without caching: $0.30 per turn
With caching: $0.375 first time, then $0.03 per turn

Over 10 turns, that's roughly 90% savings on input cost—not to mention lower latency, since less recomputation means the first token arrives faster.

3. Ordering: The More Stable, The Earlier

Since caching relies on prefix matching, the ordering of prompt content is critical. Anthropic's best practice:

1. Front: System instructions and tool definitions—fixed and shared across all sessions. 2. Second: Project docs (CLAUDE.md)—shared within a project. 3. Third: Current session context—valid only for this conversation. 4. Last: Chat messages—growing turn by turn, with only the latest message added each round.

One-line summary: the less likely something is to change, the earlier it goes.

4. Three Pitfalls: One Small Detail Breaks the Whole Chain

Pitfall 1: Embedding the current time in fixed instructions—it changes every second, killing the cache.
Pitfall 2: Storing tool definitions in unordered containers (e.g., Python dict or set)—request ordering varies, prefixes don't match.
Pitfall 3: Changing tool parameters—even one modified field invalidates the entire prefix cache.

5. Updating Information: Send a Message, Don't Touch the Prompt

Information goes stale during execution (wrong built-in time, modified files). Updating the system prompt directly triggers cache invalidation—expensive for users.

Claude Code instead passes updates as the next conversation message: it quietly inserts a <system-reminder> tag into the next user message or tool result. This tiny trick perfectly preserves the cache.

6. Don't Switch Models or Tools

Switching to Haiku for easy questions and back to Opus for hard ones sounds sensible—but caches are bound to the model. Switching invalidates all accumulated cache, and rebuilding it often costs more than just letting Opus answer the easy question. Claude Code's policy: one model throughout the main conversation.

Need a smaller model? Delegate to a sub-task with its own independent context and cache. The main model writes a task handoff summary, the sub-task executes it, and only the result returns to the main conversation.

A note for API relay/proxy operators: caches are isolated per account. Mixing an account pool tanks hit rates—you lose money and the accounts.

7. Cache-Safe Forking for Context Compression

Long conversations eventually fill the context window, requiring compression into a summary. But if the compression call uses a different system prompt or omits tool definitions, its cache chain diverges from the main conversation—you pay twice.

Anthropic's solution, Cache-Safe Forking: the compression request must use exactly the same system instructions, user context, and tool definitions, with the main conversation's messages as history, appending the compression instruction as a new user message at the end. From the backend's perspective, it's nearly identical to the previous request—only the final instruction is new cost. Also reserve a compression buffer ahead of time so the window isn't full before you start.

8. Plan Mode and Lazy Loading

In Plan mode, the model doesn't act immediately on complex requests—it first analyzes, decomposes, and lists needed tools. This state change isn't done by modifying the system prompt (which would break the cache) but simulated via a special tool call. The system prompt never changes; the cache stays intact.

Similarly, lazy loading loads tool definitions on demand rather than all upfront, keeping the static prefix minimal and hit rates high.

9. Takeaways for Your Own Agents

1. Order prompts: system instructions → tool definitions → reference docs → conversation history 2. Update information via messages, never by editing the system prompt 3. Never switch models or tools mid-conversation 4. Use sub-agents for tasks needing different models 5. Use cache-safe forking when compressing context 6. Monitor cache hit rate like server uptime

Claude Code was designed around prompt caching from day one. If you're building the next great agent, start from this rule.

References:

Anthropic blog: Lessons from building Claude Code: Prompt caching is everything
easy-learn-ai project: prompt-cache interactive talk site