LatentMoE: How Mixture-of-Experts Makes a Key Paradigm Leap in 2026

Forum topic · ✨步子哥 · 2026-07-25

Summary

LatentMoE is an evolution of Mixture-of-Experts (MoE) architecture adopted in early 2026 by two very different models: NVIDIA's 120B Nemotron 3 Super and Moonshot AI's 2.8T Kimi K3. Traditional MoE routes tokens to experts whose weight matrices scale with the model's hidden dimension d (typically 4096-7168), making memory reads and All-to-All communication expensive; activating more experts linearly amplifies these costs. LatentMoE instead projects tokens into a compressed latent space ℓ (about d/4) before routing, performs expert computation there, and projects back. This cuts weight reads and communication roughly 4x, freeing budget to activate ~4x more experts at equal inference cost. NVIDIA reports that a standard MoE would need ~350B extra parameters to match LatentMoE accuracy, and LatentMoE can run up to 3.5x faster at accuracy parity, with only ~9% projection overhead. Nemotron 3 Super pairs the design with Mamba-2 layers and MTP speculative decoding (2.2x GPT-OSS-120B throughput), while Kimi K3 scales to 896 experts with Top-16 routing and a Quantile Balancing load-balancing method, achieving ~2.5x scaling efficiency over Kimi K2. Together with linear sequence models and low-precision training, LatentMoE is emerging as a new default for next-generation MoE architectures.

LatentMoE: How MoE Makes a Key Paradigm Leap in 2026

In the first half of 2026, NVIDIA released the 120B Nemotron 3 Super, and Moonshot AI released the 2.8T Kimi K3. Despite the huge gap in scale, both models made the same architectural choice — Latent MoE. This article uses these two models to unpack what exactly changed.

1. What problem did MoE run into?

Over the past few years, MoE became the mainstream path for scaling large models: few activated parameters, many total parameters, controlled compute, high capability. DeepSeek-V4, Kimi, and Qwen3 all follow this route. But as models grew, a hidden structural problem emerged.

When a token is routed to an expert, the system must read that expert's weight matrix from memory — sized d × m, where d is the model's hidden dimension (typically 4096–7168 in modern large models). In multi-node distributed deployment, each routing step also requires an All-to-All communication whose volume is proportional to d. When you activate more experts to improve quality, both costs scale linearly. This is the structural dilemma of traditional MoE: you want better quality, but the price is too high.

NVIDIA's researchers analyzed this thoroughly: to tackle latency and throughput bottlenecks simultaneously, the only effective lever is shrinking the hidden dimension d. But d cannot be shrunk arbitrarily — feature representation needs sufficient dimensionality, or quality collapses. Hence the LatentMoE idea: don't shrink d directly; instead, project tokens into a smaller latent space ℓ before entering the experts, compute there, then project back.

2. How LatentMoE works

Three steps:

1. Token x (dimension d) is compressed via a learnable down-projection W↓ into a low-dimensional latent space ℓ (e.g., 1024, about a quarter of d) 2. Routing and expert computation happen in the ℓ dimension 3. Expert outputs are projected back to d via an up-projection W↑

The core benefit comes from this compression ratio. Expert weights shrink from d×m to ℓ×m — 4x less memory reading, and 4x less All-to-All traffic. What to do with the savings? Deploy more experts and activate more experts. At equal inference cost, the model can consult roughly 4x more experts, exponentially expanding the expert-combination space, enabling finer-grained knowledge routing and stronger capability.

A notable detail: routing gates, shared experts, and attention layers stay at the full d dimension, because they are not the bottleneck and aren't worth compressing.

3. Does it actually work? The data says yes

The natural question: does compressing into latent space hurt accuracy? NVIDIA's LatentMoE paper gives a clear comparison. At equal parameter counts, LatentMoE achieves higher accuracy than the standard Transformer MoE baseline on all evaluated tasks — reasoning, knowledge, code — and the result holds in Hybrid Mamba-Attention MoE architectures too. At equal parameters and equal inference cost, LatentMoE consistently beats standard MoE.

Key figures:

350B — extra parameters a standard MoE would need to match LatentMoE's accuracy
3.5× — LatentMoE's maximum inference speedup at accuracy parity
9% — measured overhead of the two projections, far smaller than the parameter cost of catching up

LatentMoE isn't free — it adds a linear projection before and after expert routing. But the measured overhead is about 9%, far less than the parameter cost for a standard MoE to match accuracy.

4. Two teams, different practices

Nemotron 3 Super: efficiency first

512 experts, Top-22 activation, latent dimension ℓ=1024 (a d/4 compression). Versus typical standard MoEs at similar scale (usually Top-8), that's nearly 4x more experts activated at the same inference cost. NVIDIA also added: Mamba-2 layers for main sequence modeling (linear time, making 1M context practical), a few Transformer attention layers as "global anchors," and shared-weight Multi-Token Prediction heads for speculative decoding (average acceptance length 3.45, above DeepSeek-R1). Result: inference throughput 2.2× GPT-OSS-120B and 7.5× Qwen3.5-122B, at roughly equal accuracy.

Kimi K3: scale first

K3 pushes it further: 896 experts, Top-16 activation, sparsity ratio 56:1. At this extreme sparsity, expert load balancing becomes the core stability issue. K3 proposes Quantile Balancing — deriving expert allocation directly from router score quantiles instead of heuristic updates, removing sensitive hyperparameters and making stable training at 2.8T parameters possible. The payoff: roughly 2.5× scaling efficiency over Kimi K2.

5. Two routes, same direction

| Dimension | Nemotron 3 Super | Kimi K3 | |---|---|---| | Total parameters | 120B | 2800B | | Experts / Top-K | 512 / 22 | 896 / 16 | | Sparsity ratio | 23:1 | 56:1 | | Load balancing | Aux-Loss-Free | Quantile Balancing | | Sequence modeling | Mamba-2 | Kimi Delta Attention | | Inference acceleration | MTP speculative decoding | Quantization-aware training (MXFP4) | | Core goal | Efficiency first | Scale first |

NVIDIA's route is "efficiency intelligence" — maximal capability in minimal activated parameters, suited to high-concurrency, low-latency serving. Kimi's route is "extreme scale" — LatentMoE unlocked training feasibility at 2.8T, with Quantile Balancing keeping training healthy at that scale. But the underlying logic is identical: compress the expert computation dimension and convert the savings into a larger expert-combination space.

6. Looking ahead

LatentMoE solves a concrete engineering bottleneck but has room to grow. The current compression ratio is fixed; an adaptive ratio — preserving more dimensions for information-dense tokens, compressing simple tokens harder — is a natural next step. On routing, Quantile Balancing turns load balancing from a hyperparameter-tuned black box into a deterministic statistical process; generalizations like multi-head routing or content-based dynamic Top-K are worth experimenting with. On the training side, NVIDIA observed ~7% zero-gradient parameters late in NVFP4 training — an emergent "structured sparsity" under low-precision training that architecture design could deliberately exploit.

Longer term, the combination of LatentMoE and linear sequence models (Mamba, KDA) makes "1M-token context + sparse activation" productizable rather than experimental. This sets a clear foundation for next-generation MoE: linear sequence models for long-range dependency, LatentMoE for knowledge capacity, and low-precision training for compute cost — three layers stacking into what is becoming the industry's new default.

*Sources: Nemotron 3 Super Technical Report (NVIDIA, 2026.04) · Kimi K3 Tech Blog (Moonshot AI, 2026.07)*