When a 1.6-trillion-parameter model learns to "let go"—how DeepSeek V4 tames 1M-token long text with 1/10 the memory.
*Source: Commit d9b875d (easy-learn-ai, 2026-04-25)*
Imagine reading a 1-million-word novel. A human reader will clearly remember only a few dozen key points—the protagonist's name, major plot twists, foreshadowing. Nobody memorizes every punctuation mark on every page.
But until recently, that's exactly how AI read long texts.
Traditional Transformers store every token in a short-term memory called the KV cache while processing long text. It's like photocopying every page of that novel and taping it to your walls—by the end, your room is covered in a million sheets.
DeepSeek V4 did something counterintuitive: it decided to forget most of the content and remember only what truly matters.
From 83.9 GiB to 9.62 GiB
At launch, DeepSeek published a striking figure: at a 1M-token context, the KV cache shrank from 83.9 GiB (previous generation) to 9.62 GiB.
Roughly 10x.
This isn't simple "zip-file" compression—it's a hybrid attention architecture called CSA/HCA.
Imagine attending a 100-person conference. The traditional approach: memorize every sentence everyone says and be able to recite any exchange afterward. Your mental load grows exponentially with attendance.
CSA (Compressed Sparse Attention) works like this: you listen closely to the people near you (sliding-window attention), keep only a "meeting atmosphere summary" for distant attendees (compressed KV streams), and there's a shared whiteboard of meeting themes everyone can see (shared KV).
HCA (Hierarchical Cross Attention) goes further: different levels of "listeners" attend at different granularities. Junior staff focus on execution details, middle managers on departmental coordination, executives on strategic direction. Information flows across levels, but isn't dumped wholesale on everyone.
Why This Matters
"Saving some memory—what's the big deal?"
In AI engineering, memory is everything.
When the KV cache drops from 84 GB to 10 GB, it means:
- Long-text tasks that once needed professional servers can run on a consumer-grade GPU
- API costs fall dramatically (DeepSeek V4 Flash is priced at $0.14 per million input tokens)
- Real-time applications become viable (lower latency means smoother interaction)
- NVIDIA claims 150+ TPS agent interactions on Blackwell Ultra
- Inference platforms like vLLM, Together, and Baseten published multi-GPU benchmarks on launch day
- Huawei's Ascend 950 + CANN announced compatibility—seen as a key step for China's "independent" compute stack
- CSA/HCA architecture highlights: https://substack.com/redirect/0fe9c80a-0feb-4fd8-aacb-7a35946d5e7e
- DeepSeek V4 technical report: https://substack.com/redirect/a14b3fde-deb5-4993-a38d-32e2cebe559a
- Training-scale analysis: https://substack.com/redirect/159dde94-efc3-43d8-989c-e10262d9c2f3
The deeper implication: "ultra-long context" goes from an expensive gimmick to a technology that can be deployed at scale. Previously, a 1M-token context was a toy only a few labs could afford. Now an ordinary developer with a few hundred dollars of hardware can experience it.
The Open-Source World Responds
DeepSeek V4's release triggered a chain reaction:
The community reaction is even more interesting: someone ran DeepSeek V4 Flash on a Mac with 256 GB of RAM; someone else calculated that a monthly API bill for everyday search plus coding could come to just a few dimes.
This isn't "yet another model release." It's redefining the standard of "usable."
Back to the Metaphor
Remember the 1-million-word novel?
What DeepSeek V4 teaches AI is not how to "remember harder," but how to "forget smartly."
Humans survive information overload not through infinite memory, but through selective attention—knowing what's worth keeping and what can be released. That's essentially what CSA/HCA gives AI.
From 83.9 GiB to 9.62 GiB, what's being compressed isn't just memory, but our fixation that "intelligence must equal scale."
Sometimes subtraction is the real progress.
---
Further reading (technical details)