Prompt Cache Explained: Why Prefix Caching Can Cut LLM Costs by 90%

Forum topic · 小凯 · 2026-05-05

Summary

This article explains prompt caching in large language model inference, based on an easy-learn-ai project that visualizes the concept with 15 interactive animation chapters. It covers why the prefill phase re-encodes the entire conversation history on every turn, and how prompt caching stores the KV computation of a fixed prefix so identical prefixes skip repeated work. Key numbers: cached input costs 10% of the normal price, first-time cache writes cost 25% extra, default TTL is 5 minutes (1-hour option available), and the minimum cacheable content is 1024 tokens (4096 for newer models). The article details best practices from Anthropic engineers: order prompts from most static (system instructions, tool definitions) to most dynamic (chat messages), avoid embedding timestamps in fixed instructions, keep tool definitions ordered and stable, use Plan Mode via injected system messages instead of changing tool sets, apply lazy loading with placeholder tool stubs, and delegate subtasks to isolated contexts. It concludes that Anthropic considers cache hit rate an infrastructure-level metric essential to Claude Code's viability.

Source: Commit 515b759 — feat(prompt-cache): added prompt cache feature module

When AI Learns "Photographic Memory": Why Prompt Cache Can Save 90% of Costs

Ever wondered why the longer you chat with Claude, the slower it responds and the thicker your bill gets?

The answer lies in an old engineering saying: "Cache rules everything." In the AI Agent era, this hasn't become outdated — it's now a matter of life and death for whether a system can run at all.

Prompt Cache discussed today isn't the browser cache you clear with a click. It's a precision mechanism inside LLM inference systems, designed to tackle the most expensive and time-wasting step in human-AI conversations.

---

1. Repetitive Labor: The AI Starts From Scratch Every Turn

Imagine you've had 20 turns with a coding assistant — fixing bugs, refactoring, writing tests. When you send the 20th request, what happens behind the scenes?

The model must re-encode the entire previous 19 turns — system instructions, tool definitions, all history messages — from scratch. It computes attention weights and KV values token by token before it can even start answering.

Those 19 turns are identical to the last round, yet the model re-inventories the whole shelf like an obsessive librarian. This is the most expensive, time-consuming part of LLM inference: the Prefill phase — converting input text into internal representations.

20 turns of conversation, the same content computed 19 times over. Wasteful, right?

---

2. How Prompt Cache Works: Fast-Forward Through What's Already Done

The solution in one sentence: you mark a breakpoint in the request; the backend stores the encoding results from the start up to that breakpoint. Next time the prefix is identical, it's reused directly, skipping repeated computation.

It's like rewriting your table of contents and earlier chapters every time you write a paper. With caching, you fast-forward through what's already written and only write the new part.

The keyword is "prefix matching." Cache matching isn't based on semantic similarity — it's byte-level prefix consistency. If the beginning of your new request is exactly identical to the last one, that portion's computation can be reused.

---

3. The Key Numbers: Absurd Savings

A few hard numbers explain why Anthropic treats cache hit rate as an infrastructure-level metric — on par with server uptime. A drop in hit rate triggers on-call alerts and a full incident response process.

Cache hits cost one-tenth of the price — 90% savings
First-time writes cost 25% more — you pay 1.25x to store the result
Default TTL is 5 minutes, auto-renewed with each request at no extra cost; a paid 1-hour tier is available
Short content can't be cached: generally at least 1024 tokens, 4096 for newer models

A concrete example with a 100k-character long conversation on Claude Sonnet:

Without caching: $0.30 per turn
With caching: $0.375 the first time, then only $0.03 per turn

Over 10 turns, roughly 90% of input costs are saved. Latency drops too — the more you skip recomputing, the faster the first token appears.

---

4. How to Order Your Prompt: The More Static, the Earlier

Since caching relies on prefix matching, the ordering of prompt components is critical.

Anthropic's best practice, top to bottom:

1. System instructions + tool definitions — fixed, shared across all sessions 2. Project documents (e.g., CLAUDE.md) — shared within a project 3. Current session context — valid only within this conversation 4. Chat messages — grows turn by turn, each turn only appends the last message

One rule: the less likely something is to change, the earlier it goes.

Like organizing a desk: rarely-touched reference books at the bottom, this week's materials in the middle, today's drafts on top.

---

5. Three Pitfalls: One Small Detail Breaks the Whole Cache Chain

Pitfall 1: Embedding the current time in fixed instructions. It changes every second, the prefix never matches, and the cache is dead. Correct approach — put time updates in conversation messages, never in system instructions.

Pitfall 2: Storing tool definitions in unordered containers. Python dicts or sets serialize in different orders per request, breaking the prefix. Fix: use ordered containers (lists), or sort explicitly before sending.

Pitfall 3: Changing one field in a tool parameter. Even a single changed field invalidates the entire prefix cache. That's why in Claude Code's design, once the tool set is fixed, it's essentially never modified mid-session.

---

6. Advanced Techniques: Plan Mode, Lazy Loading, Subtask Forking

Plan Mode: Switch States via Messages, Not Instructions

Claude Code has a "planning mode" where the model only thinks, not executes. The intuitive approach would be removing execution tools on entry and adding them back on exit. Anthropic didn't do this — touching tools breaks the cache.

Instead, they keep all tools unchanged and add two special tools — "enter planning" and "exit planning." How is the "no execution in planning mode" constraint conveyed? By injecting a system message into the conversation.

Key distinction: system instructions are fixed (inside the cached prefix); conversation messages are fluid (don't affect the prefix). The tool set never changes, the cache stays valid.

Lazy Loading: Placeholders First, Load on Demand

Claude Code may integrate dozens of external tools. Defining them all fully? Too much space. Adding/removing on demand? Breaks the cache.

The compromise is lazy loading: start with lightweight placeholders — the model sees only tool names, not full parameter definitions. When the model actually needs a tool, it fetches the full definition via "tool search."

The prefix always contains only lightweight placeholders, so the cache stays stable. Like a library index: browse the catalog first, then fetch the book — no need to move every book onto your desk.

Subtasks: Send Interns to Separate Desks

When a smaller model needs to do work, delegate a subtask. Subtasks have their own isolated context and cache, so they don't pollute the main conversation's cache chain.

Concretely: the main model writes a handoff document condensing the context, passes it to the subtask, and only receives the result back. You don't let an intern sit at your desk using your computer — you assign them a separate machine with a clear task description, and they send results back when done.

---

7. Cache as Infrastructure: No Cache, No Claude Code

Anthropic's internal claim: without caching, there is no Claude Code.

AI coding assistants like Claude Code involve long conversations — dozens of turns per session. Every turn re-sends the full context; recomputing from scratch each time would explode latency and cost.

The higher the cache hit rate, the more users can do at the same price. Anthropic treats it not as a performance optimization but as a cornerstone of user experience — hit rates directly affect how generous usage quotas can be for paying customers.

---

8. Looking Back: Every Lesson Says the Same Thing

All 7 lessons boil down to one principle — cache is prefix matching.

Any change at any point in the prefix invalidates everything after it. So: don't touch instructions, don't touch tools — one touch breaks the whole chain. Same for switching models, accounts, or starting fresh sessions.

This looks like cache optimization, but it's also a design philosophy — first accept the constraint you can't avoid, then build the entire system around it.

---

> This project comes from easy-learn-ai, which demonstrates the full Prompt Cache knowledge system across 15 interactive animation chapters, each with narration and visuals, playable directly in the browser. Reference: Anthropic engineers' sharing on prompt cache design best practices for Claude Code.