MiniMax Sparse Attention: Block-Level Top-K for Affordable Million-Token Contexts
Forum topic · 小凯 · 2026-06-24
Summary
MiniMax (MiniMax) introduces Sparse Attention (MSA), a minimalist two-branch architecture that slashes long-context attention cost. Built on Grouped Query Attention, MSA uses a lightweight index branch to score and select the top-k key-value blocks (block size 128, k=16, budget 2,048 tokens) for each query, then runs exact softmax attention only on the selected blocks. This block-level dynamic sparsity delivers a 28.4x reduction in per-token attention compute at 1M context length, plus 14.2x prefill and 7.6x decoding speedups on H800 GPUs. Training stability is maintained via KL divergence alignment between index and main branches, gradient detachment, and intra-group block sharing. MSA powers the open-source MiniMax-M3 model (428B total / 23B active sparse-MoE parameters, native multimodal, 1M context, SWE-Bench 80.5, supported in vLLM and SGLang).
Key points
- Core idea: MSA keeps full attention quality but restricts each query to a dynamic Top-K selection of 128-token key-value blocks, yielding a fixed 2,048-token KV budget regardless of total context length.
- Architecture: Two branches built on GQA — a lightweight index branch that scores blocks and a main branch that runs exact softmax attention only on selected blocks.
- Why block-level, not token-level: Block granularity yields regular, contiguous memory access patterns that map cleanly onto GPU tensor cores, avoiding the irregular indexing that defeats most token-level sparse schemes.
- GPU co-design: An exp-free Top-K kernel, a KV-outer sparse attention path that fuses sparsity into MMA tiling, and 128-token-aligned memory transfers deliver the observed wall-clock speedups.
- Training stability: KL loss between index and group-averaged main-branch distributions, detached index-branch gradients, and shared block selection across heads within a GQA group.
- Performance on MiniMax-M3 (109B / 428B total / 23B active sparse-MoE): 1/28.4x per-token attention compute at 1M context, 14.2x H800 prefill, 7.6x H800 decoding, with quality on par with the dense GQA baseline.
- Benchmark scores: SWE-Bench Verified 80.5, OmniDocBench 91.6, VideoMME 85.4, BrowseComp 83.5; inference supported in vLLM and SGLang.
- GQA group specialization: Different groups can attend to different long-range regions (e.g., code structure vs. natural-language comments vs. cross-file dependencies).
- Limitations: Hard-coded block size Bk=128 may miss fine-grained signals; Top-K selection can drop critical tokens; KV-cache for 1M contexts remains large and is mitigated with MXFP8 quantization.
References
- Lai et al. (2026). MiniMax Sparse Attention. MiniMax. arXiv:2606.13392.
- MiniMax-M3 model: https://huggingface.co/MiniMaxAI/MiniMax-M3
- MSA inference kernel: https://github.com/MiniMax-AI/MSA
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178208070