Nemotron 3 Super: NVIDIA's Efficiency Revolution with LatentMoE and Native NVFP4 Training

NVIDIA has released Nemotron 3 Super, a 120B-parameter Mixture-of-Experts (MoE) model that runs with only 12B active parameters. Beyond scale, its defining contribution is fusing efficiency and accuracy through architecture and training innovations, marking an "efficiency-first" stage for open large models.

Architecture: Hybrid Mamba-Transformer MoE

The model alternates three layer types:

• Mamba-2 layers: linear-time state-space layers handling most sequence processing, keeping memory manageable even with contexts up to 1M tokens.
• Transformer attention layers: inserted at key depths for precise global retrieval, compensating for the SSM's weaker needle-in-a-haystack recall.
• MoE layers: 512 experts with Top-22 routing, providing 120B-parameter knowledge capacity at 12B active compute, enabling low latency for multi-agent workloads.

The design philosophy: Mamba for efficiency, attention for precision, MoE for capacity.

Innovation 1: LatentMoE

Standard MoE routing at full hidden dimension becomes a compute bottleneck as models scale. LatentMoE projects tokens into a low-rank latent space before expert computation, then projects back. This allows activating four experts at the cost of one, greatly increasing specialization granularity (e.g., separating Python-syntax experts from SQL-logic experts) without extra inference cost.

Innovation 2: Multi-Token Prediction (MTP)

Instead of predicting one token per forward pass, the model predicts multiple future tokens simultaneously:

• Better reasoning: forcing the model to internalize longer-horizon dependencies improves chain-of-thought robustness and accuracy.
• Native speculative decoding: multi-token predictions act as drafts that can be verified in parallel, yielding up to 3x end-to-end speedup in structured generation (code, tool calls).

The MTP heads use a shared-weight multi-head design, minimizing parameters and keeping predictions consistent across steps.

Innovation 3: Native NVFP4 Pre-training

Rather than training in BF16 and quantizing afterward, Nemotron 3 Super performs most floating-point multiply-accumulate operations in NVFP4, NVIDIA's 4-bit floating format optimized for Blackwell GPUs. Training under 4-bit constraints from the start yields mathematically stable training and accuracy on par with BF16, while delivering roughly 4x inference speedup over FP8 on Blackwell hardware with no post-hoc quantization step.

Pre-training Data and Scale

• 25 trillion tokens total (more than GPT-OSS-120B's 15T), in two phases: 80% (20T) for broad diversity, 20% (5T) for high quality and benchmark accuracy.
• Targeted synthetic data for code concepts, algorithms, formal logic, and economics, with decontamination; synthetic algorithm data reportedly added 1–2 points on HumanEval/MBPP.

Post-training

Supervised fine-tuning is followed by large-scale multi-environment reinforcement learning across 21 environment configurations (multi-step tool use, software engineering, terminal operation), with over 1.2 million rollouts using the NVIDIA NeMo Gym and NeMo RL frameworks — producing strong agentic reasoning capabilities.

Performance

Accuracy (vs. comparable open models):

| Benchmark | Nemotron 3 Super | GPT-OSS-120B | |---|---|---| | MMLU-Pro | 83.73 | 81.00 | | SWE-Bench Verified | 60.47 | 41.9 |

Additional results: GPQA 82.7, AIME25 90.21. It also leads GLM-4.5-Air-Base and Ling-flash-Base-2.0 on the Artificial Analysis intelligence index.

Throughput: at 8K input / 64K output, Nemotron 3 Super delivers 2.2x the throughput of GPT-OSS-120B and 7.5x that of Qwen3.5-122B.

Openness

NVIDIA released not just weights but pre- and post-training datasets and full recipes, including over 10T tokens of pre-training data, 15 RL environments, and complete evaluation/fine-tuning pipelines — enabling private deployment and customization without black-box APIs.

Conclusion

Nemotron 3 Super demonstrates that scale growth need not come with proportional efficiency loss. Via LatentMoE, MTP, native 4-bit training, and multi-environment RL, it points toward a future focused on intelligence per FLOP and knowledge efficiency per parameter rather than raw parameter counts — opening a new chapter for open-source large models.