LatentMoE: How MoE Made a Key Paradigm Leap in 2026
In the first half of 2026, NVIDIA released the 120B Nemotron 3 Super, and Moonshot AI released the 2.8T Kimi K3. Despite their vastly different scales, both models made the same architectural choice: Latent MoE. This article uses these two models to unpack what exactly changed.
1. What Problem Did MoE Hit?
MoE has become the mainstream path for scaling large models: fewer activated parameters, more total parameters, controllable compute, high capability. DeepSeek-V4, Kimi, and Qwen3 all follow this route.
But as models grow, a hidden structural problem emerges. Every time a token is routed to an expert, the system must read that expert's weight matrix of size d × m from memory (where d is the hidden dimension, typically 4096–7168 in modern models). In multi-node distributed deployment, each routing step also requires All-to-All communication proportional to d.
Activating more experts (for quality) linearly amplifies both costs. NVIDIA's researchers concluded that to address both latency and throughput bottlenecks, the only effective lever is shrinking d—but d cannot be shrunk freely without collapsing model quality.
LatentMoE's insight: don't shrink d directly—project tokens into a smaller latent space ℓ before the experts, compute there, and project back.
2. How LatentMoE Works
Three steps:
1. Token x (dimension d) is compressed via a learnable down-projection W↓ into latent space ℓ (e.g., 1024, one quarter of d)
2. Routing and expert computation happen in dimension ℓ
3. Expert outputs are projected back to d via up-projection W↑
The core gain comes from this compression ratio: expert weights shrink from d×m to ℓ×m, cutting memory reads ~4× and All-to-All traffic ~4×. That 4× saving is reinvested into deploying and activating ~4× more experts at the same inference cost, exponentially expanding the expert-combination space for finer-grained knowledge routing.
Notably, the gating network, shared experts, and attention layers all stay at full dimension d, since they are not the bottleneck.
3. Does It Actually Work? The Data
Per NVIDIA's LatentMoE paper, at equal parameter counts LatentMoE beats standard Transformer MoE on all evaluated tasks (reasoning, knowledge, code), with the same result reproduced on Hybrid Mamba-Attention MoE architectures.
- 350B: extra parameters a standard MoE would need to match LatentMoE's accuracy
- 3.5×: maximum inference speedup for LatentMoE at accuracy parity
- 9%: measured overhead of the two added projections—far smaller than the cost for standard MoE to catch up
- Adaptive compression: the current fixed ratio (
d/ℓpreset) could become adaptive—preserving more dimensions for information-dense tokens, compressing simple tokens harder. - Routing: Quantile Balancing turns load balancing from a hyperparameter-tuned black box into a statistically deterministic process; generalizations like multi-head routing and content-based dynamic Top-K are natural next experiments.
- Training: NVIDIA observed ~7% zero-gradient parameters late in NVFP4 training—arguably emergent structured sparsity under low-precision training that architecture design could deliberately exploit.
4. Two Teams, Two Practices
Nemotron 3 Super — Efficiency First
512 experts, Top-22 activation, latent dimension ℓ=1024 (≈ d/4). Compared to typical standard MoE (Top-8), that's nearly 4× more active experts at equal cost. Key companions: Mamba-2 layers for sequence modeling (linear complexity, making 1M context practical), a few Transformer attention layers as "global anchors," and shared-weight Multi-Token Prediction heads for speculative decoding (average acceptance length 3.45, surpassing DeepSeek-R1). Result: 2.2× the inference throughput of GPT-OSS-120B and 7.5× that of Qwen3.5-122B, at comparable accuracy.
Kimi K3 — Scale First
K3 pushes further: 896 experts, Top-16 activation, 56:1 sparsity. At this extreme sparsity, load balancing becomes critical for training stability. K3 proposes Quantile Balancing—deriving expert assignment directly from router score quantiles without heuristic updates or sensitive hyperparameters, enabling stable training at 2.8T parameters. Result: roughly 2.5× scaling efficiency over Kimi K2.
5. Comparison
| Dimension | Nemotron 3 Super | Kimi K3 | |---|---|---| | Total params | 120B | 2800B | | Experts / Top-K | 512 / 22 | 896 / 16 | | Sparsity ratio | 23:1 | 56:1 | | Load balancing | Aux-Loss-Free | Quantile Balancing | | Sequence modeling | Mamba-2 | Kimi Delta Attention | | Inference speedup | MTP speculative decoding | Quantization-aware training (MXFP4) | | Core pursuit | Efficiency first | Scale first |
NVIDIA's route is "efficiency intelligence"—maximum capability in minimal activated parameters, suited to high-concurrency, low-latency serving. Kimi's route is "extreme scale"—LatentMoE unlocked 2.8T-parameter training feasibility, with Quantile Balancing keeping training healthy at that scale.
> The underlying logic is identical: compress the expert computation dimension and convert the savings into a larger expert-combination space.
6. Looking Ahead
---
Sources: Nemotron 3 Super Technical Report (NVIDIA, 2026.04) · Kimi K3 Tech Blog (Moonshot AI, 2026.07)