Caching in LLMs: How DeepSeek Slashed API Prices Through Disk-Backed Prompt Caches

Key points

• KV Cache (inference layer): stores K and V matrices from each attention step so the next token only computes its own Q/K/V, dropping complexity from O(n²) to O(n). No Q Cache exists — historical Q vectors are unused by future steps.
• Prompt Cache (API layer): vendors hash the request prefix on the server; matching prefixes skip prefill. Anthropic charges 0.1× input price on hits, OpenAI 0.25–0.5×, DeepSeek 0.1×. Minimum cache block is 1024 tokens; TTL is typically 5–10 min (Anthropic) or 10–30 min disk-persistent (DeepSeek).
• DeepSeek's disk cache breakthrough: MLA compresses KV Cache to 7–10% of standard size, fitting on cheap distributed disks instead of GPU HBM. A 128K-token prompt drops from ~13 s to ~0.5 s first-token latency on hits (26×). DeepSeek V4-Pro hits 1/120 of base input price on cache reads; V4-Flash hits 1/100. Auto-enabled, 95%+ hit rate observed.
• Cost inversion for prompt design: with DeepSeek, more context is cheaper than less — 1,000 cached input tokens cost ~$0.000028. Adding few-shot examples or full project docs barely changes the bill.
• Token resellers' hidden margin: middlemen charge users the full (uncached) rate while paying the cached upstream rate. Tactics include hiding prompt_cache_hit_tokens fields, prefix pollution (injecting unique markers to bust hits), and rate multiplier fraud. DeepSeek's transparent disk caching squeezes this arbitrage.
• Claude Code source-code patterns: a SYSTEM_PROMPT_DYNAMIC_BOUNDARY separates static prefix (cacheable) from dynamic tail (uncacheable), flagged with a DANGEROUS_ prefix. The YOLO classifier and main query engine share the same prefix to amortize cache writes. autoCompact.ts folds history before it grows too long; serializing and resuming a session can subtly shift the prefix and bust the cache.
• CLAUDE.md as cache anchor: keep it under 50 lines, put commands, conventions, file paths — no timestamps, no weekly updates, no session-specific notes. Treat it as an index, not documentation.
• Measured impact: Claude Code production runs report 92% hit rate and 81% cost reduction; 20,000-token system prompt over 50 turns costs $0.30 vs $3.00 without cache.

Structured findings

1. Why KV Cache exists

Each token generation requires Q·Kᵀ/√d_k over all prior tokens. Without caching, step n recomputes K and V for tokens 1..n−1, yielding O(n²) FLOPs. Caching prior K and V matrices reduces this to O(n). At GPT-3 175B scale, each token holds ~20 KB of KV state; 1,000 tokens ≈ 20 MB per sequence, scaling linearly with batch size.

2. Prompt Cache mechanics

• Cache key = hash of prefix starting at token 0; only exact prefix matches hit.
• Best request layout: System Prompt → Tools → CLAUDE.md → cache_control breakpoint → History → User Input → Tool Results.
• Anti-patterns: timestamps, random IDs, reordered tool definitions, mid-session model switches (cache is model-bound).

3. DeepSeek pricing matrix (per 1M tokens, May 2026)

| Model | Input (miss) | Input (hit) | Output | Hit discount | |---|---|---|---|---| | DeepSeek V4-Flash | $0.28 | $0.0028 | $0.42 | 1/100 | | DeepSeek V4-Pro | $1.74 | $0.0145 | $3.48 | 1/120 | | GPT-5.5 | $5.00 | $0.50 | $30.00 | 1/10 | | Claude Opus 4.7 | $5.00 | $0.50 | $25.00 | 1/10 |

Official data: 56.3% of input tokens hit the disk cache within 24 hours.

4. Reseller fraud anatomy

• Charge user full rate, pay upstream cached rate, pocket the spread.
• Hide prompt_cache_hit_tokens / prompt_cache_miss_tokens fields in responses.
• Inject unique prefixes so every user request has a different prefix → cache never hits.
• Example: 89M tokens/day, real cost $2.50, billed cost $25, margin $22.50/day per developer.

5. Claude Code engineering rules

• Keep the system-prompt static section byte-stable across sessions.
• Compact early per subtask; do not resume old sessions (serialization drift breaks prefix hash).
• Share prefix across components (classifier + query engine) so the cache write is amortized.
• CLAUDE.md should index where information lives, not embed it.

6. Cost outcome (Claude Opus example)

| Scenario | Input tokens | Effective cost | |---|---|---| | No cache | 1,800,000 | $9.00 | | Default (70% hit) | 540,000 | $2.70 | | Optimized (92% hit) | 280,000 | $1.40 |

Optimized cost = 15.6% of uncached cost (84% saved).

Practical rules for developers

1. Keep the prefix byte-stable — caching requires it. 2. Compact early and per subtask — late compaction pays full price. 3. On DeepSeek, add context rather than trim it; cached tokens are nearly free.

References

• Wilson Wu: Understanding KV-Cache
• DeepSeek API docs: Disk context cache
• yeasy: Claude guide — prompt caching
• Aliyun: Claude Code Prompt Cache deep dive
• 21 Jingji: Token resellers report