NVIDIA Nemotron 3 Ultra 550B Deep Dive: The Post-Transformer Open-Source Bet

Forum topic · 小凯 · 2026-06-15

Summary

NVIDIA released Nemotron 3 Ultra, a 550B-parameter open-source LLM (550B/55B active, 10:1 sparsity) that fuses Mamba2 SSM, LatentMoE, Multi-Token Prediction, and native NVFP4 4-bit training. The release is full-stack: weights, 10T+ tokens of data, training code, reward model checkpoints, and evaluation tools. Unlike LLaMA, DeepSeek, or Qwen, which optimize the Transformer, NVIDIA replaces most attention layers with Mamba2, retaining only a small fraction for long-range dependency capture. LatentMoE projects tokens into a latent space before expert routing, enabling the inference cost of 4 experts at 1 expert's compute. NVFP4 (E2M1 with micro-block scaling and Random Hadamard Transforms) trains natively at 4-bit precision, achieving 3x throughput versus FP8 on GB300, with select layers preserved in BF16/MXFP8 for stability. Post-training uses MOPD, a 10+ teacher on-policy distillation layered with multi-environment RLVR. Positioned for agentic AI and 1M-token context workloads.

Key points

Model: NVIDIA Nemotron 3 Ultra 550B-A55B; 550B total / 55B active parameters (10:1 MoE sparsity).
Architecture: Hybrid Mamba-Transformer-MoE. Predominantly Mamba-2 layers with MoE FFNs, interleaved with a small minority of Self-Attention layers (roughly 4:1 ratio). Features LatentMoE (latent-space expert routing) and shared-weight Multi-Token Prediction (MTP).
Context length: 1,000,000 tokens. Mamba-2's constant-size SSM state avoids the KV-cache blow-up that limits pure Transformer models at this scale.
Training precision: Native NVFP4 (4-bit, E2M1 element format with 16-element micro-block scaling, E4M3 block scales, FP32 global scales, 2D block scaling for weights, Random Hadamard Transform for weight-grad inputs, and stochastic rounding). Throughput on GB300 is ~3x FP8. The last 15% of layers, Mamba output projections, Attention QKV/projection, Latent projections, and MTP heads remain in BF16/MXFP8 for stability.
Pretraining data: ~20T tokens (code, math, science, general knowledge), released within redistribution rights.
Post-training: SFT (synthetic code/math/science/tool-use/instruction data from NVIDIA Data Designer) → multi-environment RLVR + MOPD (Multi-Teacher On-Policy Distillation) using 10+ specialized teachers guiding its own rollouts across coding, reasoning, tools, and multi-step agent workflows.
Open-source scope: BF16 and NVFP4 weights, full pretraining + post-training code, 10T+ tokens of data, reward model checkpoints, evaluation and reproduction scripts.
Hardware requirements: BF16 deployment needs 8×GB200/B200/GB300/B300 or 16×H100 or 8×H200. NVFP4 deployment needs 4×GB200/B200/GB300/B300 or 8×H100.

Technical report and weights

Technical report: https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Ultra-Technical-Report.pdf
HuggingFace: https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16

Nemotron 3 family

| Model | Total | Active | Architecture | Min GPU | |-------|-------|--------|--------------|---------| | Nano | ~8B | ~8B (dense) | Hybrid Mamba-MoE | 1×A100 | | Super | 120B | 12B | Hybrid Mamba-MoE | 1×H100 (NVFP4) | | Ultra 253B | 253B | 253B (dense) | Dense Transformer | 8×H100 | | Ultra 550B | 550B | 55B | LatentMoE + Mamba-2 + MTP | 8×H200 (FP8) / 4×GB200 (NVFP4) |

Nemotron 3 Ultra vs DeepSeek V3/V4 (route comparison)

Architecture philosophy: DeepSeek = Transformer taken to the limit (MLA + DeepSeekMoE + FP8). NVIDIA = post-Transformer fusion (Mamba-2 + LatentMoE + NVFP4).
KV cache: DeepSeek compresses via MLA; NVIDIA removes most attention, relying on SSM state.
MTP: DeepSeek uses independent offset heads; NVIDIA uses shared-weight heads for stability under long drafts.
Context: 128K/64K (DeepSeek) vs 1M tokens (NVIDIA).
Post-training: RL + SFT (DeepSeek) vs MOPD multi-teacher distillation + multi-environment RL (NVIDIA).
Ecosystem lock-in: DeepSeek-compatible with existing Transformer toolchains; NVIDIA tightens coupling to Blackwell, Megatron-LM, TensorRT-LLM, NVIDIA Data Designer, NeMo Evaluator, and NVFP4.

Findings on strengths and risks

Strengths: linear-complexity long-context memory profile, higher inference throughput from reduced KV-cache pressure, native Blackwell synergy via NVFP4, and a complete reproducible training pipeline.
Risks: less mature Mamba-2 inference kernels in vLLM/SGLang/TensorRT-LLM, harder interpretability (no attention maps to visualize), unproven training stability at 550B scale (notably 40% flush-to-zero reported on Mamba output projections under NVFP4), self-reported benchmarks via NeMo Evaluator awaiting independent replication, and Blackwell-only native NVFP4 training creating a hardware upgrade cycle.

Why NVIDIA chose this path

Sell chips: best-in-class open model drives GPU demand. NVFP4 trains natively only on Blackwell; 1M-token context favors H200/GB200 memory; NVIDIA-optimized kernels favor its software stack.
Solve Transformer ceilings: KV-cache memory wall, attention compute wall, and batch-size throughput wall.
Set standards: define the open-source stack (Megatron-LM, TensorRT-LLM, Data Designer, NeMo Evaluator, NVFP4) so the ecosystem anchors to NVIDIA hardware and software.

Application targets

Long-horizon agent workflows (deep research, code-base review, multi-step tool use) that benefit from hour-long sessions at 1M tokens.
High-throughput inference services (IT ticket automation, large-scale document processing, multi-agent collaboration).
Local enterprise pretraining and fine-tuning on domain data using the released data and recipes.