DeepSeek V4 — Squeezing a 1M-Token "Memory Palace" onto a GPU
*English translation of a zhichai.net forum post.*
Sherlock Holmes famously used a "memory palace" to store every clue, name, and date in a mental building. DeepSeek V4 does something similar — except its palace holds 1 million "memory fragments," compressed from a space that would normally require 84GB of VRAM down to under 10GB.
What does 1M context mean?
A million tokens is roughly enough to hold an entire novel plus annotations. Earlier large models handled only 8K–128K context — a few chapters at most. Beyond that, models behave like someone who finished a book but forgot the beginning.
DeepSeek V4 raises the ceiling to 1M tokens: an entire codebase, a textbook, months of chat history, or a novella can fit into a single conversation.
But a bigger memory palace costs more to maintain.
KV Cache: the model's short-term memory notes
LLMs "remember" context via the KV Cache (Key-Value Cache), one of the largest inference memory costs. For 1M tokens, a traditional architecture's KV cache can balloon past 80GB — requiring multiple high-end GPUs.
DeepSeek V4's hybrid CSA/HCA attention architecture redesigns how these "notes" are stored:
- Shared KV: references like "he," "it," "this" point to one entry instead of being duplicated.
- Compressed KV streams: unimportant content is condensed; important content stays detailed.
- Sparse attention: CSA (Compressed Shared Attention) lets the model decide "which drawers stay open, which get locked."
Not just compression — redesigned memory
Training used 32–33T tokens (~1e25 FLOPs) with a parameter/data ratio of about 1:20 — the model has "read" far more than its own capacity.
FP4+FP8 mixed-precision storage: MoE experts use FP4, while attention, normalization, and routing use FP8 — like an accountant keeping quick ledgers for daily entries and precise books for critical figures.
Huawei Ascend: another path
DeepSeek V4 is technically compatible with Huawei Ascend 950 + CANN, meaning it can run outside the NVIDIA CUDA ecosystem. Ascend supply is currently about a quarter of H100 volumes, but DeepSeek has publicly said that once Ascend 950 supernodes scale in the second half of the year, V4 Pro pricing will drop noticeably — seen as a milestone for China's self-sufficient compute stack.
Benchmarks: open-source ceiling, but chatty
Per Artificial Analysis, V4 Pro Max scored 52 (+10 over V3.2), second only to Kimi K2.6. In GDPval agent tasks, V4 Pro leads all open models. Caveats: hallucination rate on AA-Omniscience is still 94%, and the same benchmark run emitted 190M tokens (Flash: 240M) — cheap per-token pricing doesn't mean cheap total tasks if the model is verbose.
V4 Flash is the opposite extreme: 284B total / 13B active parameters, slightly weaker than Pro, but API pricing of just $0.14/$0.28 per million input/output tokens. Community estimates put monthly API bills for everyday search + coding at only a few dimes.
Conclusion
DeepSeek V4 isn't the biggest or highest-scoring model, but it makes "very long memory" affordable. When a GPU can hold a novel's worth of context and local deployment drops from lab-grade to small-company affordable, AI use cases expand from Q&A to companion reading, long-form fiction, and whole-codebase analysis.
It's like swapping a notepad that holds one chapter for an index-card box that holds an entire library. The design of the card box is the real engineering art worth studying.
---
Source commit: d9b875d (easy-learn-ai)