On July 1, NVIDIA released Nemotron-Labs-TwoTower, a block-autoregressive diffusion language model. The weights are open-sourced under the NVIDIA Nemotron Open Model License and hosted on Hugging Face at nvidia/Nemotron-Labs-TwoTower-30B-A3B-Base-BF16.
Technical Details
- Backbone: Built on Nemotron-3-Nano-30B-A3B (MoE hybrid with interleaved Mamba-2 / self-attention / MoE layers), frozen and used as the context tower
- New module: A diffusion denoiser tower, trained on 2.1T tokens (vs. 25T for base pretraining)
- Two-tower division of labor:
- AR context tower (frozen): performs causal processing of the prompt and committed tokens, producing per-layer KV caches and Mamba-2 states, preserving the backbone's autoregressive capability
- Diffusion denoiser tower (trained): refines noisy blocks—bidirectional attention within blocks, causal attention to past clean blocks across blocks
- Connection: layer-aligned cross-attention—denoiser layer *i* cross-attends to context tower layer *i*, a richer "multi-scale access" than prior schemes that only broadcast the final hidden state
- Total parameters: ~60B across both towers; ~3B activated per token per tower; MoE with 128 routable experts, 6 activated + 2 shared
- Layers: 52 per tower (23 Mamba-2 + 6 self-attention + 23 MoE)
generate_mask_diffusion(): full two-tower diffusion (2 GPUs, ~59GB/GPU BF16)generate_mock_ar(): mock autoregressivegenerate_ar(): pure autoregressive (single 80GB GPU)- AI infrastructure teams: evaluate TwoTower for batch synthesis, offline embeddings, nightly jobs—if 2.42× throughput holds, GPU-hour bills could drop 50%+
- Training data teams: diffusion LLMs may reshape synthetic data economics—a dataset costing 100 GPU-days on AR might take 42 GPU-days
- Architecture researchers: frozen-backbone-plus-denoiser is a reusable engineering template
- NVIDIA customers: TwoTower serves as a reference workload for H100/H200 clusters (NVLink, memory bandwidth, BF16 stability)
- Competing labs: against closed Gemini Diffusion, Mercury, and LLaDA, open weights set an open baseline for the diffusion LLM track
- Code and math quality drops are material: HumanEval 79.27 → 75.58 (-3.69), MATH-500 84.40 → 80.60 (-3.80), GSM8K 92.49 → 90.14 (-2.35). For coding use, a 3.7-point drop is real—data generated with this degradation could pollute downstream models
- Full diffusion needs 2 GPUs (~59GB/GPU BF16), unfriendly to single-GPU deployment; mock-AR and AR modes run single-GPU but aren't diffusion
- Base model only: no instruction tuning or alignment, so direct use will produce off-target answers
- Quality degrades sharply past 3× throughput: the paper notes throughput beyond 3× comes with larger quality loss, so 2.42× is a sweet spot, not a ceiling—whether real deployments can push toward 3× is unknown
- Licensing caveats: the NVIDIA Nemotron Open Model License is custom—not Apache 2.0 or MIT—so commercial compliance needs legal review
- Anomalous arXiv ID: MarkTechPost cites arxiv 2606.26493, worth verifying against the original paper
- MarkTechPost technical walkthrough: https://www.marktechpost.com/2026/07/01/nvidia-releases-nemotron-labs-twotower/
- Hugging Face model collection: https://huggingface.co/collections/nvidia/nemotron-labs-twotower
- arXiv paper: https://arxiv.org/pdf/2606.26493
Benchmark results (γ=0.8, S=16, 2×H100 BF16):
| Task | Nemotron-3-Nano-30B-A3B (AR) | TwoTower (diffusion) | |------|-----|-----| | MMLU (5-shot) | 78.56 | 78.24 | | MMLU-Pro (5-shot CoT EM) | 62.59 | 60.93 | | ARC-Challenge (25-shot) | 91.72 | 92.66 | | WinoGrande (5-shot) | 76.09 | 76.09 | | RACE (0-shot) | 88.90 | 88.90 | | HumanEval (0-shot) | 79.27 | 75.58 | | MBPP-Sanitized (3-shot) | 74.71 | 74.28 | | GSM8K (8-shot) | 92.49 | 90.14 | | MATH-500 (4-shot) | 84.40 | 80.60 | | MMLU Global Lite | 73.97 | 73.94 | | MGSM (8-shot avg) | 80.80 | 80.40 | | Quality retention | 100% | 98.7% | | Throughput | 1.0× | 2.42× |
Three inference modes (one checkpoint supports all):
Tuning knob: the confidence threshold γ—lowering γ commits more tokens per step, raising throughput but reducing quality. The paper shows γ=0.8 is the sweet spot: 1.3% quality loss for 2.42× speed.
Analysis
1. Diffusion LLMs go from concept to engineering open source. Diffusion language models aren't new—Google Gemini Diffusion, Inception's Mercury, and Meta's LLaDA all exist—but all were closed-source or academic demos. TwoTower is the first open-weight industrial diffusion LLM, which matters more than any specific benchmark number.
2. "Frozen AR backbone + trained denoiser" is an elegantly frugal solution. Prior diffusion LLMs required full re-pretraining—a cost no team would pay to bet on an unproven paradigm. TwoTower freezes a 25T-token AR backbone and trains only a 2.1T-token denoiser, i.e. 8.4% of full pretraining cost (2.1T / 25T). This template lets any existing AR model be "diffusion-ized" cheaply.
3. Where is a 1.3% quality loss for 2.42× throughput commercially valuable? The direct answer is batch synthetic data generation—RLHF data, agent trajectory synthesis, document summarization distillation. The indirect answer is throughput-first API workloads insensitive to latency (overnight batches, offline embeddings). For latency-sensitive interactive use cases (real-time chat, coding agents), AR remains dominant—throughput gains are dominated by user waiting time and can't be monetized.
Impact on AI coding tools: synthetic training data is a core cost of coding-model iteration. If stable, TwoTower's diffusion mode could produce equivalent-quality data with 1/2.42 the GPU hours—cutting training data costs by nearly 60%. For teams doing open-source coding models, this offers a low-cost path to scale training data.
Strategic significance for NVIDIA: TwoTower runs BF16 on 2×H100, demonstrating that "H100 clusters are the best platform for diffusion LLM training + inference" as AMD MI300, Intel Gaudi, Cerebras CS-3, and others compete for inference workloads.