Nemotron 3 Super: NVIDIA's Efficiency Revolution - Deep Dive into the First LatentMoE + NVFP4 Open Large Model
NVIDIA recently released Nemotron 3 Super, a 120B-parameter mixture-of-experts (MoE) model that runs with only 12B active parameters. Rather than simply stacking parameters, it is the first open model to deeply fuse efficiency and accuracy through architectural innovation, marking a new "efficiency-first" stage for open large language models. This article provides a deep analysis of its technical report.
Architecture Overview: Hybrid Mamba-Transformer MoE
Nemotron 3 Super's core is a hybrid Mamba-Transformer MoE design with three alternating layer types:
- Mamba-2 layers: Handle the bulk of sequence processing. As state-space models, they have linear-time complexity, keeping memory and compute manageable even at a 1M-token context window.
- Transformer attention layers: Interleaved at key depths to ensure precise global retrieval. Pure SSMs can struggle to find a "needle" in a large context; attention layers guarantee high-fidelity recall of critical details.
- MoE layers: Expand effective capacity via sparse activation. The model has 512 experts with Top-22 routing, keeping 12B active parameters while retaining the knowledge capacity of a 120B model.
- Stronger reasoning: Predicting several tokens forces the model to internalize longer-range dependencies, making chain-of-thought generation more consistent and accurate.
- Built-in inference acceleration: MTP acts as native speculative decoding - its multi-token drafts are verified in parallel, yielding up to 3x end-to-end speedup in structured generation tasks such as code generation and tool calling.
The design philosophy: Mamba for efficiency, attention for accuracy, MoE for capacity.
Innovation 1: LatentMoE - Latent-Space Compression, 4x Expert Parallelism
In standard MoE, the router passes full-dimension token embeddings to experts, and the router becomes a scaling bottleneck. LatentMoE projects tokens into a low-rank latent space before routing, performs expert computation in the compressed dimension, then projects back.
The payoff: more experts at the same cost. Because tokens are compressed, the model can activate four experts for roughly the compute of one, achieving 4x expert parallelism. Finer-grained experts also allow more precise task routing (e.g., separating Python syntax experts from SQL logic experts), improving specialization without extra computation.
Innovation 2: Multi-Token Prediction (MTP)
Instead of predicting one token per forward pass, Nemotron 3 Super predicts multiple future tokens at each position:
Innovation 3: Native NVFP4 Pretraining
Rather than the conventional "high-precision training, then quantize" pipeline, Nemotron 3 Super performs most floating-point multiply-accumulate operations in NVFP4, NVIDIA's 4-bit floating-point format optimized for the Blackwell architecture. Training natively at 4-bit makes the model mathematically stable under low precision, so it retains near-BF16 accuracy while achieving up to 4x inference speedup versus FP8 on Blackwell GPUs. Deployment requires no separate quantization step, significantly cutting inference costs - particularly valuable for long-running autonomous agent workloads.
Pretraining Data and Scale: 25 Trillion Tokens
The model was pretrained on 25 trillion tokens (more than GPT-OSS-120B's 15T), split into two stages:
1. Stage 1 (80%, 20T tokens): diversity and broad coverage. 2. Stage 2 (20%, 5T tokens): high-quality data and benchmark accuracy.
The corpus includes carefully designed, decontaminated synthetic data for code concepts, algorithms, formal logic, and economics, yielding genuine gains (e.g., 1-2 point improvements on HumanEval and MBPP) rather than memorization.
Post-Training: SFT and Multi-Environment RL
After supervised fine-tuning (SFT), the model underwent large-scale reinforcement learning across 21 distinct environment configurations, covering multi-step tool use, software engineering, and terminal operations. Using NVIDIA's NeMo Gym and NeMo RL frameworks, training exceeded 1.2 million environment rollouts - unprecedented scale and diversity among open models, powering strong agentic reasoning.
Performance: Accuracy and Efficiency Verified
Accuracy highlights:
| Benchmark | Nemotron 3 Super | GPT-OSS-120B | |---|---|---| | MMLU-Pro | 83.73 | 81.00 | | SWE-Bench Verified | 60.47 | 41.9 |
Additional results: GPQA 82.7 (top tier among open models), AIME25 90.21 (excellent math reasoning), and a leading Artificial Analysis Intelligence Index versus GLM-4.5-Air-Base and Ling-flash-Base-2.0.
Efficiency: at 8K input / 64K output, inference throughput is 2.2x GPT-OSS-120B and 7.5x Qwen3.5-122B - critical for agents that repeatedly send long histories and tool outputs. This stems from the hybrid Mamba-Transformer MoE design, with LatentMoE and MTP further accelerating inference.
Openness: A New Bar for Open Large Models
NVIDIA released not only the weights but the full pretraining and post-training datasets and recipes: over 10 trillion tokens of pretraining data, 15 RL training environments, and complete evaluation/fine-tuning pipelines. Developers can customize, optimize, and self-host the model without black-box API dependencies - significant for research, community improvement, and privacy-sensitive deployments in finance and healthcare.
Conclusion: Efficiency First, a New Chapter for Open Models
Nemotron 3 Super shows that scaling parameters need not sacrifice efficiency. Through LatentMoE, MTP, and native NVFP4 training, it achieves more capability with less compute - essential for building reliable, cost-controlled autonomous agent systems. It offers the community a clear technical roadmap: hybrid architectures, sparse activation, low-precision training, and multi-environment RL. Future models will be judged less on raw parameter counts and more on intelligence per FLOP and knowledge per parameter - and Nemotron 3 Super opens that chapter.