> Paper: MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens > arXiv: 2603.23516 > Team: multi-institution (incl. Tianqiao Chen / Tianqiong Tian Gong)
1. The Problem: Why AI Has "Goldfish Memory"
Human lifelong memory holds roughly 200-300 million tokens, while mainstream LLMs effectively handle only ~1M tokens of context — a two-order-of-magnitude gap. This is not a compute shortage; it is rooted in the math of attention: standard Transformer attention is O(n²), so doubling context quadruples compute. 1M tokens already strains engineering teams; 100M tokens is impossible under the traditional framework.
Existing approaches each have their own trap:
- Linear attention / state compression (e.g., RWKV, Titans): fast, but accuracy degrades sharply as context grows.
- RAG / external retrieval: requires complex pipelines and heuristic hyperparameters, and is not end-to-end trainable.
- Long-context models (e.g., Qwen2.5-1M): brute-force scaling, very expensive inference, and performance collapses beyond training length.
- Routing Key: kept in GPU memory for fast query matching (~56GB for 100M tokens).
- Content KV: kept in CPU memory, lazily loaded to GPU only when selected.
- 100M: max supported context tokens
- <9%: performance drop from 16K → 100M
- 94.84%: NIAH accuracy at 1M tokens
- 2×A800: minimum GPU config for 100M-token inference
- 56GB: GPU memory for 100M-token Routing Keys
- 169GB: total compressed cache footprint incl. CPU offload
- 64K: training context length (extrapolated to 100M at inference)
- O(n): inference complexity (vs O(n²) standard attention)
MSA's authors spotted a blind spot: why must memory capacity be coupled with reasoning compute? Like the human brain — fixed size, but nearly unlimited memory.
2. Core Idea: Decouple "Remembering" from "Reasoning"
MSA does not make the model "process" 100M tokens; it lets the model precisely retrieve the few thousand most relevant tokens from 100M at any time for reasoning. Three key designs:
2.1 Document-wise RoPE: each document "lives in its own time zone"
Standard positional encodings break when models trained on ~10 concatenated documents must handle 10,000 documents with positions in the millions. MSA restarts position numbering at 0 for every document. The insight: semantic relevance does not depend on a document's absolute position in the corpus. The model trains on 64K contexts (few documents) and extrapolates at inference to 100M tokens (many documents), because the positional "receptive field" never changes.
2.2 KV Cache Compression: turning a library into index cards
Each document's KV cache is compressed into an "index card":
Flow: query → similarity against all Routing Keys → Top-k documents selected → load only those Content KVs from CPU → attention computed. Complexity is O(n) because the query interacts only with compressed keys, not raw tokens.
2.3 Memory Interleave: a shuttle for multi-hop reasoning
Single-shot Top-k retrieval can miss documents needed for cross-document inference. Memory Interleave performs iterative retrieve-generate-retrieve: intermediate reasoning results become new queries — like humans thinking while flipping through references.
3. The Numbers: Not Just "Runs", But "Runs Steadily"
Core claim: performance drops less than 9% from 16K to 100M tokens.
3.1 Needle-in-a-haystack (NIAH)
| Model | 32K | 128K | 256K | 1M | |-------|-----|------|------|-----| | MSA | 98.77% | 97.5% | 96.2% | 94.84% | | Qwen3-4B (backbone) | ~95% | ~70% | 48.16% | 24.69% | | Qwen2.5-14B-1M | ~98% | 89.97% | collapse | collapse |
3.2 Long-context QA (MS MARCO)
| Context length | MSA score | Drop | |----------------|-----------|------| | 16K | 4.023 | - | | 100M | 3.669 | 8.8% |
GPT-4.1 and DeepSeek-V3.2 show clear degradation and cliff-edge collapse past 512K on comparable tests.
3.3 Hardware feasibility
100M-token inference needs only 2× NVIDIA A800 GPUs (80GB). Routing Keys are distributed in GPU memory; Content KV sits in CPU memory with asynchronous pipelined loading. No H100/H200, no 8-GPU node — a standard cloud server suffices.
4. Why MSA Works: Three Overlooked Design Choices
1. End-to-end trainable: unlike RAG's fragmented pipeline (chunking, indexing, retrieval, reranking, generation), MSA puts everything in a differentiable framework optimized jointly by gradients — the compressor learns "what to keep for downstream tasks." 2. Train-inference decoupling: trained on 64K, inferred on 100M. Document-wise RoPE makes this legitimate: the model never "processes 100M tokens" — it always processes a concatenation of a few dozen documents, just precisely selected from the corpus. 3. Retrievable, not perfect compression: the compressed KV need not reconstruct the original; it only needs to "stand in the right spot" in vector space to be found by queries — freeing the compressor to use any lossy scheme.
5. Limitations and Open Questions
1. Strong cross-document dependencies: complex structural links across documents (e.g., a table in A combined with a chart in B) may exceed single Top-k retrieval; Memory Interleave mitigates but does not solve this. 2. Dynamic memory modification: the framework assumes a static memory (encode once, query many); incremental updates remain unsolved. 3. Integration with MoE/sparse architectures: validated on dense Qwen3-4B; combining with MoE+CSA/HCA-style designs is a natural next step.
6. A Feynman-Style Verdict
Is it a breakthrough? Yes — not because of the 100M number itself, but because it proves long context can avoid the O(n²) brute-force path. MSA establishes a new paradigm: memory ≠ context. Context is what is currently being processed; memory is an on-demand retrieval library. Decoupled, a model can keep constant inference cost while gaining near-unlimited memory.
Cargo cult? Not quite — solid math, end-to-end training, reproducible benchmarks. But beware the trap of competing on "how many hundred-million tokens" while ignoring retrieval quality. 100M tokens with inaccurate retrieval is worse than 100K tokens with precise hits.
Would I use it? For a "lifelong memory assistant" — recording all user conversations, documents, and emails with precise on-demand recall — MSA is currently the most viable architecture, attachable as a plugin to existing models.