How Prompt Cache Works in LLMs: 10 Counter-Intuitive Rules from Claude Code

Forum topic · 小凯 · 2026-05-23

Summary

Prompt caching eliminates the redundant prefill cost in every conversational turn, but it only works under one rigid constraint: exact prefix matching. This guide walks through 10 practical lessons distilled from Anthropic's Claude Code architecture. Hit tokens cost roughly 10% of fresh tokens; cache entries default to a 5-minute rolling TTL and require at least 1024 tokens (4096 on newer models). A well-ordered prompt stacks stable content first—system instructions and tool definitions at the bottom, project docs above, session context in the middle, and chat messages on top. Common pitfalls—embedding timestamps in system prompts, using unordered tool collections, or modifying tool schemas—break the cache chain instantly. Switching mid-session to a smaller model also invalidates accumulated cache, so sub-tasks with isolated contexts are preferred. Plan Mode keeps the tool set fixed and signals state through conversation messages instead of system edits. Cache hit rate is treated as production-grade infrastructure at Anthropic, and account-pooled relay services destroy hit rates.

A Waste Worth Fixing

Every message you send to Claude or GPT triggers a full prefill: the model re-encodes the system prompt, tool definitions, conversation history, and new input from scratch. In a 20-turn conversation, the first turn is re-encoded 20 times. Prompt Cache exists to kill this redundancy.

How It Works: Prefix Matching

Mark a breakpoint in your request. The backend stores the encoding from start to breakpoint. On the next request, if the prefix is byte-identical, the cached prefix is reused and only the new suffix is computed. Similar is not enough—*identical* is the rule. One changed character invalidates everything after it.

The Savings Are Real

Cached tokens cost roughly 10% of fresh-token price.
First write costs ~25% extra; subsequent hits save ~90%.
Default TTL: 5 minutes, auto-renewed on each hit.
Minimum cacheable length: 1024 tokens (older models) or 4096 tokens (newer models).

Example: a 100k-token conversation on Claude Sonnet runs about $0.30/turn without caching, $0.375 first turn with caching, then $0.03/turn. Over 10 turns, input cost drops by roughly 90%, and TTFT (time-to-first-token) improves noticeably because less has to be recomputed.

Anthropic monitors cache hit rate as production infrastructure—on par with uptime. Drop in hit rate triggers a formal incident response.

The Four-Layer Desk

Order matters because of prefix matching. Anthropic's recommended stack:

1. System instructions + tool definitions — fixed, shared across sessions. 2. Project docs (e.g., CLAUDE.md) — shared within a project. 3. Session context — valid only in this conversation. 4. Chat messages — grows turn by turn.

Rule of thumb: *the less often something changes, the earlier it goes.*

Three Cache-Breaking Traps

Trap 1 — Embedding time in the system prompt. A Today is {datetime.now()} field changes every second and breaks the prefix. Fix: keep system instructions frozen and inject updates into the conversation via <system-reminder> tags.

Trap 2 — Unordered tool containers. Python dict or set iteration order can vary between requests. Use ordered list.

Trap 3 — Mutating tool schemas. Adding one parameter or changing a field type invalidates the entire prefix chain, since tool definitions live at layer 1.

Counter-Intuitive: Don't Switch Models

Caches are model-bound. Switching to a smaller model mid-session wipes all accumulated cache, and rebuilding it often costs more than just letting the large model answer the easy question.

Claude Code's solution: keep the main conversation on one model. For lightweight work, dispatch sub-tasks with their own isolated context and cache. Write a concise handoff briefing, let the sub-task run, and return only the result—like giving an intern a separate machine instead of swapping them into your seat.

Plan Mode Without Breaking Cache

Intuition says: enter plan mode → remove execution tools; exit plan mode → add them back. Anthropic does the opposite. The tool set stays frozen, and two extra tools (enter_plan_mode, exit_plan_mode) handle transitions. The "no execution" constraint is delivered as a system message inside the conversation flow, not as a system-prompt edit. Bonus: the model can self-trigger plan mode when it senses a complex task.

Lazy Loading and Cache-Safe Forking

Lazy loading: group tools by scenario; load only what's needed so the prefix stays shorter and more stable.
Cache-safe forking: when compressing, summarizing, or branching a conversation, the fork must share the same prefix—same model, same system prompt, same tool definitions—or it rebuilds from zero.

A Note for Relay Services

Cache is account-isolated. Pooling accounts across requests tanks the hit rate. Auto-rotating accounts every couple of messages has the same effect.

The One Rule Behind Everything

All ten lessons reduce to one constraint: cache equals prefix matching.

Any change anywhere in the prefix invalidates everything after it.
Use messages, not instructions, for dynamic content.
Don't swap tools or models mid-session.
Monitor hit rate like uptime.
Forks must share the parent's prefix.

This looks like a caching optimization, but it is really a design philosophy: *identify the immovable constraint first, then architect the system around it.*

---

*Based on the 515b759 commit of the easy-learn-ai project, which offers an interactive, animation-driven, 15-chapter walkthrough of Prompt Cache design—inspired by Anthropic engineers' talks on Claude Code's caching architecture.*