When AI Learns to "Remember What You Said" — The Hidden Bill You're Quietly Overpaying Behind Prompt Cache
> Source commit: 515b759 | easy-learn-ai daily update
> Inspired by Anthropic engineers' best-practice sharing on Claude Code prompt cache design
---
1. The Invisible Move Costing You 90% Extra
Imagine you're a lawyer discussing a complex contract with a brilliant assistant. You've talked through nineteen rounds — background, clauses, risk analysis, revision suggestions. On the twentieth round, you add one sentence: "Can the penalty ratio in Clause 3 be changed to 2 percent?"
Then you watch this assistant re-read all nineteen previous rounds from start to finish. Not a word skipped. Starting from "Hello, I'm your legal assistant" all the way to "Clause 19 dispute resolution" — just to answer your final question, which makes up a thousandth of the whole conversation.
Wouldn't you think this person is crazy?
Yet this is exactly what most large language models do today. The more you chat with Claude, GPT, or Gemini, the higher the proportion of repeated work. After twenty rounds, nineteen rounds of content are completely duplicated — but every round, the model re-"reads" everything. That wastes time, and in API billing, time directly equals money. And you're paying every cent of it.
Anthropic's engineers did the math: for a 100k-token long conversation without caching, Claude Sonnet burns about $0.30 per turn — $6 over twenty turns. With caching enabled, the first write costs slightly more ($0.375), then each subsequent turn costs only $0.03. Ten rounds in, you save ninety percent.
This isn't an optimization. It's an order-of-magnitude cost difference between using it and not using it.
2. Caching Isn't Magic — Just a Clever "Partial Reuse"
Let's zoom into what happens behind the scenes.
Every time you send a message to an LLM, the first step is prefill — encoding all your text, from the first token to the last, into an internal representation called the KV Cache. This process accounts for the majority of inference cost and is the main source of latency.
Here's the problem: after twenty rounds of conversation, the first nineteen rounds in your twentieth request are identical to the previous turn. System prompt unchanged, tool definitions unchanged, history untouched. Yet the model recomputes from the first token, like that lawyer's assistant re-reading everything.
Prompt Cache's solution is remarkably simple:
You mark a breakpoint in the request. The backend stores the encoding result from the start up to that breakpoint. Next time, if the prefix is exactly the same, it's reused directly, skipping redundant computation.
That's it. No architecture changes, no new algorithms. Just engineers realizing: since the prefix doesn't change, why not store the computed result?
Think of it like writing a paper: without cache, you recopy the table of contents and earlier chapters each time before adding new content. With cache, the copied parts fast-forward, and you only write the new section.
3. The Numbers, Recalculated
Key figures worth remembering:
- Cache hits are priced at one-tenth of normal input cost. If regular input is $0.003 per 1K tokens, a cache hit costs $0.0003.
- The first write costs 1.25x (25% extra), but every subsequent hit saves 90%. As long as your conversation lasts more than one turn, this is always a winning trade.
- Default TTL is 5 minutes, auto-renewed by requests within the window at no extra charge. A paid 1-hour tier exists for long-lived contexts.
- There's a threshold: content that's too short can't be cached. Generally at least 1024 tokens (roughly 700+ Chinese characters), or 4096 for newer models. Short prompts don't benefit — which is why this technology mainly serves long conversations and complex Agent scenarios.
- Use messages instead of instruction changes
- Don't switch tools or models mid-conversation
- Monitor cache hit rate like you monitor uptime
- Forks must share the main conversation's prefix
- Don't touch instructions or tools — one touch, and the whole cache chain breaks
- Anthropic engineers' sharing on Claude Code Prompt Cache best practices
- easy-learn-ai Prompt Cache interactive tutorial (15 chapters)
Latency gains are equally significant: the more you skip recomputing, the faster the first token appears — the metric known as TTFT (Time To First Token). In multi-turn conversations, users notice "when does it start replying" far more than total duration. Caching directly slashes that wait.
4. Ordering Matters — What a Cache-Friendly Prompt Looks Like
Since caching is prefix matching, the arrangement of content in your prompt directly determines your hit rate.
Anthropic's best practices divide prompt content into four layers, front to back:
1. System instructions + tool definitions. Fixed, shared across all sessions. Identity, capabilities, callable tools — never touch these. 2. Project documents. E.g., CLAUDE.md, shared across conversations in the same project. Coding conventions, project structure, tech stack — relatively stable. 3. Current session context. Valid only for this conversation — currently open files, today's goals. 4. Chat messages. Grow turn by turn; only the last message is new each round.
One-line summary: the more stable something is, the earlier it goes.
It's like organizing a desk: rarely-touched reference books on the bottom shelf, this week's materials in the middle, today's drafts on top. That way you never have to rummage through the whole desk.
Conversely, if you stuff a real-time timestamp that changes every second into the bottom layer, or store tool definitions in an unordered container with different ordering each request, the prefix changes every time and cache never hits.
5. Three Pitfalls That Break the Cache Chain
Pitfall 1: Embedding the current time in fixed instructions. A system prompt containing "today is {datetime.now()}" sounds reasonable — but the prefix differs second to second, killing the cache. The correct approach: don't modify the system prompt. Push updates into the next message instead — attach a <system-reminder> block inside the next user message. The foundation (system instructions) stays pinned; the flowing water (messages) can change freely.
Pitfall 2: Tool definitions in unordered containers. dict/set produce different orderings per request, so prefixes never match. Solution: use a list with fixed ordering.
Pitfall 3: Changing even one field of a tool's parameters. Like a chain with one cracked link, everything after it falls apart. Design tools for stability — avoid changing fields when possible, prefer state fields over adding/removing parameters.
6. Don't Switch Models — A Counterintuitive Truth
You might think: route simple questions to a small model (e.g., Haiku) to save money, and switch back to a big model (e.g., Opus) for hard ones. Reasonable, right? For a 100k-token conversation whose final turn is just a formatting request, why not use a cheap model?
But the fact is: caches are bound to the model.
Switching models invalidates all accumulated caches, forcing a full rebuild — and the rebuild cost often exceeds what the big model would charge to answer that "simple question" directly.
So Claude Code's strategy is: the main conversation uses one model throughout. If you truly need a small model for light work:
Dispatch subtasks.
Subtasks have their own independent context and caches, so they don't pollute the main conversation's cache chain. The main model writes a "task handoff brief" condensing the context and sends it out. The sub-model returns only results; intermediate work stays in the subtask. Like not letting an intern sit at your desk using your computer — give them their own workstation.
Claude Code's Explore agents work this way: Haiku independently runs searches, code reading, and summarization with independent caches, leaving the main conversation untouched.
A warning for proxy/relay providers: caches are isolated per account. If you run account pools for relays, caches don't share across accounts, hit rates plummet — you make no money and lose the accounts.
7. Lazy Loading, Plan Mode, and Cache-Safe Forking
Lazy Loading: Don't register all tools at the start of a conversation. Load on demand. Since tool definitions are part of the prefix, more tools mean longer prefixes and bigger blast radius for any change. Lazy loading splits tools into on-demand modules, reducing prefix size and instability.
Plan Mode: Have the model output an execution plan first, then enter execution after user confirmation. The cache trap here: if switching from plan mode to execution mode changes system instructions or tool definitions, the cache breaks. The right way is to switch states via messages, not by modifying system instructions.
Cache-Safe Forking: When compressing, summarizing, or spawning subtasks, ensure the forked conversation shares the main conversation's prefix. The "foundation" layers — system instructions, tool definitions, project docs — must be identical; only the message layer may differ. Otherwise, forking means breaking the chain.
8. Conclusion: Prefix Matching Determines Everything
Looking back, all fifteen lessons say the same thing — cache is prefix matching.
Any change anywhere in the prefix invalidates the cache for everything after it. This is Prompt Cache's single unavoidable constraint, and all design should revolve around it:
This looks like cache optimization, but it's also a mindset: first accept the unbreakable constraint, then build the entire system around it.
In the AI Agent era, where context routinely starts at 100k tokens, Prompt Cache has gone from "optional optimization" to "mandatory." Not understanding cache design means silently overpaying 90% on every API call. Understand prefix matching, and you'll naturally write cache-friendly prompt structures.
Anthropic has done the math for you. The question is — when will you start saving?
---
References