This post, based on the easy-learn-ai project (commit 515b759), explains Prompt Caching in large language models, inspired by Anthropic engineers' best practices around Claude Code's prompt cache design. Below is a structured English rendering of the full article.
An Absurd Scenario
Imagine asking a librarian what the first chapter of *One Hundred Years of Solitude* is about. The librarian reads from page one to the end of chapter one before answering. The next day, when you ask a follow-up question about the same chapter, the librarian re-reads the whole chapter from the beginning again—because "the rule is to start from the beginning every time."
This is exactly how LLMs have long worked: in a 20-turn conversation, turn 20 re-encodes the identical history of turns 1–19 from scratch. The insight behind prompt caching: if the earlier content hasn't changed, why not let the model "take notes"?
The Secret of Taking Notes: Prompt Cache
Prompt caching marks a "breakpoint" in the input—like a bookmark. Everything before the breakpoint is encoded once and stored. If a later input has an identical prefix, the cached result is reused.
The economics: analyzing a 100k-token document over 10 turns without caching costs about $3.00 ($0.30/turn). With caching, the first turn costs $0.375 (a 25% cache-write premium), but subsequent turns cost $0.03 each—total ~$0.645, nearly 80% savings. Savings grow with more turns.
Latency: the time to first token (TTFT) drops significantly once the cache hits, because the model no longer re-reads long history.
Practical details: caching requires at least 1024–4096 tokens, entries default to a 5-minute TTL, and the TTL auto-renews while the conversation continues.
Why Anthropic Treats It as Infrastructure
Without caching, Claude Code—where multi-turn deep conversations over a codebase spanning dozens of turns are common—would be impossible: latency and cost would explode. Anthropic monitors cache hit rate like uptime; a drop triggers a SEV alert and on-call response. High hit rate → lower cost → more generous quotas → more users → a flywheel.
The single core principle: prefix matching. The cache only recognizes byte-identical beginnings; a one-character difference (even a changed space) invalidates everything from that point on. Hence the design constraint: don't change instructions, don't touch tools.
Lessons from Claude Code
1. Don't Modify System Prompts
Switching modes (e.g., "execute" → "plan") by editing the system prompt breaks the cache. Claude Code instead uses two special tools ("enter planning" / "exit planning") and inserts in-conversation system messages like "you may only think, not execute." The system prompt stays fixed inside the cache prefix; flowing conversation messages don't affect the static prefix.
2. Don't Swap Tools Mid-Conversation
Loading/unloading tool definitions changes the prefix. The compromise is lazy loading: the model initially sees lightweight placeholder entries—like a library's index card catalog—and fetches full tool definitions on demand via a special "tool search" tool. The prefix stays stable.
3. Isolate Subtasks
Subtasks (e.g., searching a codebase) generate large intermediate outputs that pollute the main context and hurt hit rates. Better: hand off via a task brief to a subagent with its own independent cache, returning only results. Also beware shared API account pools: mixed prefixes dilute hit rates, and in extreme cases an account with a too-low hit rate can get banned—a real internal lesson at Anthropic.
4. Cache-Safe Forking for Compression
Naive context compression sends history to a new request with a different system prompt and no tool definitions—never matching the main cache and paying full price. Anthropic's Cache-Safe Forking instead makes the compression request share the exact prefix (same system prompt, user context, tool definitions), appends the history plus one new user message ("compress the above into a summary"), and reserves a compression buffer so the request itself doesn't overflow the window. The only incremental cost is the final compression instruction.
Back to Fundamentals: Prefix Matching Decides Everything
1. Use messages instead of instruction changes. Messages flow; the prefix doesn't. 2. Don't switch tools or models mid-conversation. Both are part of the prefix. 3. Monitor cache hit rate like uptime. It's infrastructure, not optional. 4. Forks must share the main prefix. Compression, subtasks, rollbacks—reuse the prefix, reuse the cache.
You don't need to memorize seven rules. Just understand one thing: the cache only recognizes identical beginnings; everything else follows.
A Revolution in "Memory"
Prompt caching looks like a money-saving trick, but its real significance is that it makes long conversations possible. Without it, AI coding assistants couldn't work across dozens of turns in large codebases, and "agents" and "deep research" would remain toy demos. It addresses an ancient computing problem: how to remember the past while efficiently processing new input. Prompt caching is a cache layer for the model's memory.
Closing Thought
Every time you send a new message in a long conversation, the model asks: "Does the beginning of this match my notes?" If yes, it opens its notebook; if not, it reopens the book from page one. That judgment determines your wait time, your bill, and how far the conversation can go.
Prompt caching is AI's notebook. It lets the AI stop starting from scratch—and makes your long conversations affordable.
---
*Source commit: 515b759 · Project: easy-learn-ai*