What Happened
On July 1, NVIDIA released Nemotron-Labs-TwoTower, a block-level autoregressive diffusion language model. The weights are open-sourced under the NVIDIA Nemotron Open Model License on HuggingFace at nvidia/Nemotron-Labs-TwoTower-30B-A3B-Base-BF16.
Technical details:
- Base architecture: Built on Nemotron-3-Nano-30B-A3B (MoE hybrid backbone with interleaved Mamba-2 / self-attention / MoE layers), frozen as the context tower
- New module: A diffusion denoiser tower, trained on 2.1T tokens (vs 25T for the base model's pretraining)
- Two-tower division of labor:
- AR context tower (frozen): performs causal processing of prompts and committed tokens, generating per-layer KV caches and Mamba-2 states, preserving the backbone's autoregressive capability
- Diffusion denoiser tower (trained): refines noisy blocks—bidirectional attention within blocks, causal attention to past clean blocks across blocks
- Connection: layer-aligned cross-attention—denoiser layer *i* cross-attends to context tower layer *i*, providing "multi-scale access" that is richer than prior approaches (which broadcast only the last hidden layer)
- Total parameters: ~60B across both towers; ~3B activated per token per tower; MoE with 128 routed experts + 6 activated + 2 shared
- Layer count: 52 layers per tower (23 Mamba-2 + 6 self-attention + 23 MoE)
generate_mask_diffusion(): full two-tower diffusion (2 GPUs, ~59GB/GPU BF16)generate_mock_ar(): mock autoregressivegenerate_ar(): pure autoregressive (single 80GB GPU)- AI infrastructure teams: evaluate TwoTower for batch synthesis, offline embedding, and nightly batch workloads—if the 2.42× throughput holds, GPU-hour costs could drop over 50%
- Training data teams: diffusion LLMs may reshape synthetic data economics—a dataset taking 100 AR GPU-days might take ~42
- Architecture researchers: frozen-backbone-plus-denoiser is a reusable engineering template
- NVIDIA customers: TwoTower serves as a reference workload for validating NVLink, memory bandwidth, and BF16 stability on H100/H200 clusters
- Competing labs: Gemini Diffusion, Mercury, and LLaDA are closed or semi-open; NVIDIA's open weights set an open baseline for the diffusion LLM track
- Code and math quality drops are material: HumanEval 79.27 → 75.58 (−3.69), MATH-500 84.40 → 80.60 (−3.80), GSM8K 92.49 → 90.14 (−2.35). Data generated with this degradation could pollute downstream models
- Full diffusion needs 2 GPUs (~59GB/GPU BF16); the single-GPU AR mode is not diffusion
- Base model only: no instruction tuning or alignment; expect off-target answers without fine-tuning
- Quality degrades sharply past 3× throughput: 2.42× is a sweet spot, not a ceiling; whether production can push toward 3× is unknown
- License caveats: the NVIDIA Nemotron Open Model License is custom—not Apache 2.0 or MIT—so commercial compliance needs legal review
- Unusual arXiv ID: the cited preprint (arxiv 2606.26493) has an anomalous numbering format and should be verified against the original paper
- MarkTechPost technical write-up: https://www.marktechpost.com/2026/07/01/nvidia-releases-nemotron-labs-twotower/
- HuggingFace model collection: https://huggingface.co/collections/nvidia/nemotron-labs-twotower
- arXiv paper: https://arxiv.org/pdf/2606.26493
Benchmark results (γ=0.8, S=16, 2×H100 BF16):
| Task | Nemotron-3-Nano-30B-A3B (AR) | TwoTower (diffusion) | |------|-----|-----| | MMLU (5-shot) | 78.56 | 78.24 | | MMLU-Pro (5-shot CoT EM) | 62.59 | 60.93 | | ARC-Challenge (25-shot) | 91.72 | 92.66 | | WinoGrande (5-shot) | 76.09 | 76.09 | | RACE (0-shot) | 88.90 | 88.90 | | HumanEval (0-shot) | 79.27 | 75.58 | | MBPP-Sanitized (3-shot) | 74.71 | 74.28 | | GSM8K (8-shot) | 92.49 | 90.14 | | MATH-500 (4-shot) | 84.40 | 80.60 | | MMLU Global Lite | 73.97 | 73.94 | | MGSM (8-shot avg) | 80.80 | 80.40 | | Quality retention | 100% | 98.7% | | Throughput | 1.0× | 2.42× |
Three inference modes (one checkpoint supports all):
Tuning knob: the confidence threshold γ—lowering γ commits more tokens per step, increasing throughput at the cost of quality. The default γ=0.8 operating point trades 1.3% quality loss for 2.42× speed.
Analysis
Three dimensions make TwoTower significant:
1. Diffusion LLMs move from concept to engineered open release. Diffusion language models aren't new—Google Gemini Diffusion, Inception's Mercury, and Meta's LLaDA all explored them. But prior efforts were closed-source or academic demos. TwoTower is claimed to be the first open-weight industrial-grade diffusion LLM, and that milestone matters more than any single benchmark number.
2. "Frozen AR backbone + trained denoiser" is an engineering-smart approach. Previous diffusion LLMs required full re-pretraining—a cost no team would pay to gamble on a new paradigm. TwoTower freezes the 25T-token AR backbone entirely and trains only a 2.1T-token denoiser, about 8.4% of full pretraining cost. This template—reuse heavy existing assets, pay only a lightweight conversion cost—means any existing AR model could in principle be cheaply "diffusion-ized."
3. Where does a 1.3% quality loss for 2.42× throughput make commercial sense? The direct answer is batch synthetic data generation—RLHF data, agent trajectory synthesis, document distillation. The indirect answer is throughput-prioritized API services—nightly batch jobs and offline embeddings where latency doesn't matter but GPU hours do. Latency-sensitive interactive use cases (real-time chat, coding agents) remain AR territory, since throughput gains don't translate to perceived user wait time.
Impact on AI coding tools: synthetic training data is a core cost of coding-model iteration. If TwoTower's diffusion mode is stable, equivalent training data could be produced with 1/2.42 the GPU hours—a roughly 60% cost reduction, a direct advantage for large labs and a low-cost data expansion path for open-source coding model teams.
Strategic significance for NVIDIA: TwoTower runs BF16 on 2×H100, positioning NVIDIA hardware as the best platform for diffusion LLM training and inference as AMD, Intel, Cerebras, and others compete for inference workloads.