MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens
> Paper: MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens > arXiv: 2603.23516 > Team: multiple institutions (including Tianqiao Chen / Tianqiong)
1. The Problem: Why Is AI Stuck with "Goldfish Memory"?
Human lifelong memory holds roughly 200–300 million tokens, but mainstream large models effectively handle only about 1M tokens of context — a two-order-of-magnitude gap.
This is not a compute problem. The root cause is the mathematical nature of attention: standard Transformer attention is O(n²) — double the context and compute quadruples. 1M tokens already strains engineering teams; 100M tokens is impossible under the traditional framework.
Existing approaches each hit a wall:
- Linear attention / state compression (e.g., RWKV, Titans): fast, but accuracy degrades sharply as context grows
- RAG / external retrieval: requires complex retrieval pipelines and heuristic hyperparameters, not end-to-end trainable
- Long-context models (e.g., Qwen2.5-1M): brute-force stacking, extremely expensive inference, and performance collapses beyond the training length
- Routing Key: stored in GPU VRAM for fast query matching. ~56GB for 100M tokens.
- Content KV: stored in CPU memory, loaded to GPU only when selected — lazily.
- 100M: maximum supported context tokens
- <9%: performance drop from 16K → 100M
- 94.84%: NIAH accuracy at 1M tokens
- 2×A800: minimum GPU config for 100M-token inference
- 56GB: VRAM for 100M-token Routing Keys
- 169GB: total memory for the full compressed cache (including CPU offload)
- 64K: training context length (extrapolated to 100M at inference)
- O(n): inference complexity (vs. O(n²) for standard attention)
The MSA authors spotted an overlooked blind spot: why must memory capacity be tied to reasoning capability? Why can't a model work like the human brain — fixed capacity, but nearly unlimited memory?
2. Core Idea: Decouple "Remembering" from "Reasoning"
MSA's essence is not making the model "process" 100M tokens, but letting the model precisely retrieve the few thousand most relevant tokens from 100M at any time for reasoning.
Three key designs:
2.1 Document-wise RoPE: Every Document Lives in Its Own Time Zone
Standard positional encodings break down when a model trained on 10 concatenated documents must handle 10,000 documents at inference, with positions ranging from 0 to millions.
MSA's fix is simple but effective: position indices restart from 0 within each document. Document A's first token is pos=0; document B's first token is also pos=0.
The insight: semantic relevance does not depend on a document's absolute position in the corpus. Training uses only 64K context (a few documents); inference extrapolates directly to 100M tokens — because the positional encoding's "receptive field" is always a single document.
2.2 KV Cache Compression: Turning the Library into Index Cards
MSA compresses each document's KV cache into an "index card":
Matching flow: query → similarity against all Routing Keys → Top-k documents selected → only those k documents' Content KV loaded from CPU → attention computed.
Overall complexity is O(n) instead of O(n²), since queries interact only with compressed Routing Keys, not raw tokens.
2.3 Memory Interleave: A Shuttle for Multi-Hop Reasoning
Single-round Top-k retrieval fails when answers require cross-document reasoning (e.g., document A says "a company has financial problems," document B says "that company's CEO resigned" — both are needed to infer causation).
Memory Interleave solves this with iterative retrieve–generate–retrieve: retrieved documents enter the context, the model produces intermediate reasoning, and those intermediate results serve as new queries — like a human thinking while flipping through references.
3. The Numbers: Not Just "Runs," but "Runs Steadily"
MSA's core value is not "can handle 100M tokens" (many claim that) but less than 9% performance drop from 16K to 100M.
3.1 Needle-in-a-Haystack (NIAH)
| Model | 32K | 128K | 256K | 1M | |-------|-----|------|------|-----| | MSA | 98.77% | 97.5% | 96.2% | 94.84% | | Qwen3-4B (backbone) | ~95% | ~70% | 48.16% | 24.69% | | Qwen2.5-14B-1M | ~98% | 89.97% | collapse | collapse |
3.2 Long-Context QA (MS MARCO)
| Context length | MSA score | Drop | |---------------|-----------|------| | 16K | 4.023 | – | | 100M | 3.669 | 8.8% |
For comparison, GPT-4.1 and DeepSeek-V3.2 show clear degradation as context grows, with cliff-like collapse after 512K.
3.3 Hardware Feasibility
100M-token inference needs only two NVIDIA A800 GPUs (80GB). Routing Keys are distributed across GPU VRAM; Content KV sits in CPU memory with asynchronous pipelined loading.
No H100/H200, no 8-GPU node required — standard cloud servers work.
4. Why MSA Works: Three Overlooked Design Choices
4.1 End-to-End Trainable
RAG pipelines (chunking, indexing, retrieval, reranking, generation) optimize each stage independently. MSA puts the entire flow into a single differentiable framework optimized jointly by gradients — so the compressor learns "what information matters downstream" rather than compressing blindly.
4.2 Train–Inference Decoupling
Train at 64K, infer at 100M. Document-wise RoPE makes this legitimate: the model is never asked to "process 100M tokens" — it always processes a concatenation of a few dozen documents, just precisely selected from the 100M corpus.
4.3 Compress for Retrieval, Not for Fidelity
Traditional KV compression aims to reconstruct the original KV. MSA's compression goal is "let queries find this document precisely." The compressed KV need not be human-readable — only well-placed in vector space — freeing the compressor's design space.
5. Limitations and Open Questions
1. Strong cross-document dependencies: complex structural reasoning across documents (e.g., combining a table in A with a chart in B) may exceed single-round Top-k retrieval. Memory Interleave mitigates but doesn't fully solve it. 2. Dynamic memory updates: the current framework assumes a static memory bank (encode once, query many times). Efficient incremental updates remain unsolved. 3. Integration with MoE/sparse architectures: validated on dense Qwen3-4B; combining with MoE-style architectures is a natural next step.
6. A Feynman-Style Verdict
Is this a major breakthrough?
Yes — but not because of the 100M number itself. What matters is that it proves long context doesn't require the O(n²) brute-force path. MSA demonstrates a new paradigm: memory ≠ context. Decoupling the two gives models near-unlimited memory at constant inference cost.
Is it cargo cult?
Not entirely — solid math, end-to-end training, reproducible benchmarks. But there is a trap: people may start competing over "how many hundred-million tokens my model handles" while ignoring that retrieval quality is the core. 100M tokens with imprecise retrieval is worse than 100K tokens with precise hits.
Would I use it?
For a "lifelong memory assistant" — logging all user conversations, documents, and emails with precise on-demand recall — MSA is currently the most viable architecture, and it can attach to existing models as a plugin.