Qwen3.8-Flash-Next Deep Dive: Moving Model Capacity from Compute to Storage

Forum topic · ✨步子哥 · 2026-09-02

Summary

Qwen3.8-Flash-Next, open-sourced by Alibaba's Qwen team on August 26, 2026, is a 180B-parameter multimodal MoE model positioned as an early architectural preview of Qwen4. Only 6B parameters activate per token (a 30x sparsity ratio), while capacity is shifted into storage: a 51.2B-parameter N-gram embedding table of 20 million 2/3-gram phrases that sits outside the compute path and can be offloaded to host memory. The architecture uses a 48-layer backbone arranged as 12 blocks of 3 GDN linear-attention layers plus 1 QSA sparse-attention layer, an extremely narrow hidden dimension of 2560, 512 tiny MoE experts (10 routed + 1 shared), a 4-branch Gated Residual design, and a Muon + AdamW split optimizer. Day-0 support came from SGLang, vLLM, TensorRT-LLM, llama.cpp and others. The post verifies the 180B figure via shard-level inspection (125.7B backbone + 51.2B N-gram + ~2.6B MTP + 0.449B ViT), debunks viral claims (e.g., it cannot run unquantized on one RTX 4090), and concludes the model wins on cost but loses on fidelity versus heavier-activation rivals.

Qwen3.8-Flash-Next Deep Dive: Moving Model Capacity from "Compute" to "Storage"

*Structured summary of a long-form Chinese forum analysis (zhichai.net source post). The original article is extensive with SVG diagrams; below are its key findings, faithfully condensed.*

Key points

Positioning: Released August 26, 2026 by Alibaba's Qwen team as a multimodal MoE model and an early architectural preview of Qwen4 — analogous to how Qwen3-Next previewed the architecture later used across Qwen3.5–3.8.
Core thesis: The model shifts capacity from computation to storage — 180B total parameters but only 6B activated per token (30x sparsity), with 51B of that capacity living in a lookup table that doesn't participate in per-token matrix multiplication.

Specifications (verified against official blog, SGLang, and HF model card)

| Item | Value | |---|---| | Total parameters | 180B (125.74B backbone + 51.2B N-gram + ~2.6B MTP + 0.449B ViT) | | Activation per token | 6B | | Hidden dimension | 2560 (extremely narrow vs. 7168 for DeepSeek-line and Kimi K3) | | Layers | 48 = 12 × (3× GDN→MoE + 1× QSA→MoE), a 3:1 linear:sparse ratio | | Attention | GDN (V 48 heads / QK 16 heads, head_dim 128) + QSA (Q 24 / KV 2, head_dim 256), QSA indexer with compression ratio 4, 2048-token budget | | MoE | 512 experts, 10 routed + 1 shared, expert dim 640 | | Gated Residual | 4 branches, bottleneck rank 320 | | N-gram Embedding | 20M 2/3-gram entries at layer 2, 51.2B params (~95.4 GiB in BF16), offloadable to host memory | | Context | 262,144 native, YaRN to 1,000,000 | | Optimizer | Muon (main weights) + AdamW (embeddings/router/GR low-rank) | | License | qwen-community-1.0 (attribution requirements >100M MAU or >$20M monthly revenue) | | Pricing (Qwen3.8-Flash production) | $0.15/M input, $0.47/M output, default 1M context |

Four claimed innovations

1. Attention — GDN + QSA: Gated DeltaNet (lossy compression, fixed-size state) handles "remembering"; QSA sparse attention (12 layers) handles precise retrieval, using an MQA-style indexer to coarsely filter then read selected blocks. 2. Residual — Gated Residual: widens the residual stream into 4 gated branches. 3. Embedding — N-gram Embedding: a 51.2B "phrase dictionary" of 20M 2/3-grams; deterministic lookup, not compute. The theoretical primitive (Engram) was proposed by DeepSeek in January 2026 but not used in DeepSeek-V4 — Qwen shipped it first in production open weights. 4. Optimization — Muon + AdamW split across parameter types.

Fact-checks from the article

180B is real: third-party shard-level count (131 safetensors files, ~360 GB BF16) confirms the breakdown — but only 6B participates per-token compute.
"Runs on a single RTX 4090 without quantization" — false per the article's verification.
"Training cost is 1/9" — unverified/nuanced; flagged rather than confirmed.
"Beats same-size and larger competitors" — supported by third-party benchmarks cited (e.g., a 58.7 score surpassing DeepSeek-V4-Flash-0731's 54.4 on the referenced benchmark).

Ecosystem and deployment

Day-0 support from SGLang, vLLM, TensorRT-LLM, llama.cpp, Unsloth, KTransformers, ms-swift; an NVIDIA GB300 NVL72 deployment blog; FlagOS community added 10 chip adaptations and 8 fused Triton operators in one day — evidence of a coordinated pre-release effort.
Real deployment hinges on the N-gram table's offloadability and new kernels for GDN/QSA.

Article's verdict

The model wins on cost (extreme sparsity, tiny activation, storage-side capacity) but loses on fidelity versus competitors with far larger activation budgets (e.g., ~104B activated). The narrow 2560 hidden dimension is highlighted as the most overlooked number. Expected to form the foundation of Qwen4.