Qwen3.8-Flash-Next: Moving Capacity from "Compute" to "Storage"
> One-line conclusion: Qwen3.8-Flash-Next, open-sourced by Alibaba on August 26, 2026, is not just a new model but an architecture manifesto — with 180B total parameters and only 6B activated per token, it signals that the next step for large models is moving capacity from "compute" to "storage." It wins on cost; it loses on fidelity.
*Note: this article is a structured English summary of a long-form Chinese forum deep-dive. The source contains detailed SVG diagrams and tables; numbers below are preserved as reported in the post.*
Key points
0. Why release an early "Qwen4 preview"
- Official positioning: a multimodal MoE model and an early architecture preview of Qwen4, analogous to how Qwen3-Next previewed the architecture later used across Qwen3.5–3.8.
- Strategic motivations per the author: 1. Claim architectural discourse: the N-gram Embedding builds on the "Engram" primitive proposed by DeepSeek in January 2026 (reportedly reserved for DeepSeek V5), yet Qwen shipped it first in production-grade open weights. 2. Ecosystem pre-adaptation: SGLang, vLLM, TensorRT-LLM, llama.cpp, Unsloth, KTransformers, ms-swift offered day-0 support; NVIDIA published a GB300 NVL72 deployment blog; FlagOS reported 10 chip adaptations plus 8 new Triton fused kernels in one day — evidence of coordinated pre-release engineering. 3. Competitive hedging against DeepSeek-V4-Flash-0731 (which lifted DeepSWE from 7.3 to 54.4 via post-training); Qwen scores 58.7 with a better cost structure.
- Linear attention → DeltaNet → Gated DeltaNet
- Sparse attention → NSA → DSA (DeepSeek) → QSA
- Residual expansion → Hyper-Connections → mHC → Gated Residual
- N-gram embedding / PLE (DeepSeek's "Engram" concept, January 2026)
- Muon optimizer
- Benchmarked against models with far larger activation budgets (e.g., ~104B activated) using only 6B.
- Third-party evaluations cited; the model reportedly leads on aggregate/cost-adjusted scores (e.g., 58.7 vs DeepSeek-V4-Flash-0731's 54.4 on the cited benchmark).
- The three benchmarks it loses, per the author, "expose the shape of the architecture" — reflecting the fidelity cost of lossy linear attention and heavy sparsification.
- ❌ "Runs on a single RTX 4090 without quantization" — false; the 51B N-gram table alone is ~95.4 GiB in BF16.
- ⚠️ "Training cost is only 1/9" — unverified/misleading framing.
- ⚠️ "180B total params" — numerically accurate but potentially misleading, since only 6B is active per token.
- ⚠️ Official benchmarks may favor home turf.
- ✅ "Outperforms same-size and larger competitors" — supported by cited third-party evaluations.
- SGLang's three key optimizations enable serving; day-0 support across major inference stacks.
- Real deployment barrier remains the N-gram table's memory footprint and host-memory offload bandwidth.
- The author notes breadth of day-0 support is a marketing figure; actual usability varies.
- Wins: cost efficiency, serving price ($0.15/$0.47 per M tokens), long-context economics, ecosystem readiness.
- Loses: fidelity on tasks demanding exact recall over long contexts, versus higher-activation competitors.
- Qwen4 outlook: expect this architecture as the foundation going forward.
- One-line failure mode: lossy compression + lookup tables can miss what dense attention would have caught.
- The "1/9 training cost" claim and several benchmark details are marked as unverified in the source post.
1. Model profile
| Item | Value | Notes |
|---|---|---|
| Total params | 180B | 125B backbone + 51B N-gram + 4B MTP (+0.449B ViT) |
| Activated per token | 6B | 30× sparsity ratio |
| Hidden dim | 2560 | Extremely narrow (DeepSeek-lineage models use 7168) |
| Layer layout | 12 × (3×(GDN→MoE) → 1×(QSA→MoE)) | 36 GDN + 12 QSA layers, 3:1 |
| GDN | V 48 heads / QK 16 heads, head_dim 128 | Linear attention, fixed-size state |
| QSA | Q 24 / KV 2 heads, head_dim 256, RoPE dim 64 | Sparse global attention |
| QSA Indexer | MQA: 4 query + 1 shared key head, head_dim 128 | Compression ratio c4, 512 blocks / 2048 tokens budget |
| MoE | 512 experts, 10 routed + 1 shared | Expert intermediate dim 640 |
| Gated Residual | 4 branches, bottleneck rank 320 | 4× widened residual stream |
| N-gram Embedding | 20M 2-gram/3-gram entries, at layer 2 | 51.2B params, ~95.4 GiB in BF16, offloadable to host memory |
| Vocab | 248,320 | |
| Context | Native 262,144; YaRN to 1,000,000 | |
| Optimizer | Muon (main weights) + AdamW (Embedding/Router/GR low-rank) | |
| MTP | 1 layer, multi-step training | 4B params (third-party shard audit measured 2.607B) |
| License | qwen-community-1.0 | Attribution required above 100M MAU or $20M monthly revenue; separate license for MaaS/AI office assistants |
| Pricing (prod Qwen3.8-Flash) | $0.15 / M input, $0.47 / M output | Default 1M context |
Where "180B" comes from: 125.74B backbone + 51.20B N-gram table + ~2.61B MTP + 0.449B ViT ≈ 180B. A third-party audit (csidia) verified 131 safetensors shards, 359,999,963,128 bytes total. Only 6B participates in per-token compute; the 51B N-gram table is a deterministically addressed lookup that need not occupy GPU memory.
2. The four innovations
1. Attention — GDN + QSA: Gated DeltaNet layers act as lossy compression ("remembering," fixed-size state, KV doesn't grow); QSA sparse attention layers do precise retrieval ("looking up"), selecting 512 blocks / 2048 tokens per query via a lightweight MQA indexer — "scan the table of contents, then read the source." 2. Residual — Gated Residual: expands the residual stream from one path to four learned branches (bottleneck rank 320), extending the Hyper-Connections / mHC lineage. 3. Embedding — N-gram Embedding: a 51B-parameter "phrase dictionary" of 20 million 2/3-gram entries inserted at layer 2, offloadable to host RAM — the literal embodiment of "moving capacity to storage." 4. Optimization — Muon + AdamW: Muon for main matrix weights; AdamW for embeddings, router, and low-rank GR branches.