Key points
- One model, four modalities. Most AI agents today chain separate models (VLM for vision, ASR for speech, OCR for documents). NVIDIA's Nemotron 3 Nano Omni instead handles text, images, video, and audio in a single shared context — cutting stacked latency, broken context, multiplied cost, and cascading errors.
- Efficient architecture. Core is a Hybrid Mamba-Transformer MoE: Mamba layers for long-sequence efficiency, Transformer layers for precise reasoning, MoE routing activating only 3B of 30B parameters per task. NVIDIA claims 4x memory/compute efficiency and 4x throughput over Nemotron 2 Nano.
- Vision pipeline: C-RADIOv4-H encoder with native 1920×1080 input (no downscaling), dynamic resolution preserving aspect ratio (replacing tile-based approaches), and Conv3D temporal compression halving video tokens.
- Audio beyond transcription: Built on Parakeet-TDT-0.6B-v2; training includes music understanding, environmental sound recognition, and speaker separation.
- Efficient Video Sampling (EVS): 1080p at 1 FPS / 128 frames; 720p at 2 FPS / 256 frames; up to ~2 minutes per clip. An engineering trade-off balancing visual density against context-window budget.
- NVIDIA's MediaPerf: highest throughput across task categories, lowest cost for video tagging, and 9x throughput vs. comparable open omni-modal models.
- Demonstrated on a single B200 GPU with vLLM nightly: 32 images at 1024×1536, or 512 frames at 512×512, within one 256K-context pass.
- H Company's computer-use agent ingests full 1920×1080 screen recordings at native resolution, materially improving GUI navigation precision.
- Weights released in BF16, FP8, FP4
- "Large portion" of training data and the full training recipe published
- Compatible with vLLM, SGLang, Ollama, llama.cpp
- Commercially usable license with no additional restrictions
- Hugging Face: https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
- Technical report: https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3-Omni-report.pdf
- arXiv: https://arxiv.org/html/2604.24954v1
- Official blog: https://developer.nvidia.com/blog/nvidia-nemotron-3-nano-omni-powers-multimodal-agent-reasoning-in-a-single-efficient-open-model/
Benchmarks (per NVIDIA)
| Benchmark | Nemotron 3 Nano Omni | Qwen3-Omni | Qwen3.5-Omni | |---|---|---|---| | DailyOmni (30/60s clips) | 74.5 | 71.9 | 73.6 | | WorldSense (long video) | 55.2 | 54.0 | — | | OCRBench-V2, MMLongBench-Doc, ChartQA | Leading | — | — | | ScreenSpot / ScreenSpot-Pro | Leading | — | — |
The Nemotron 3 family
| Model | Size | Role | |---|---|---| | Nano Omni | 30B-A3B | Sub-agent perception layer, cost-optimal | | Super | 120B-A12B | Multi-agent orchestration | | Ultra | 253B | Complex enterprise planning |
The three can be composed into a full agent pipeline built entirely on NVIDIA open models.
Openness
Limitations noted by the author
1. 3B active parameters — not a general-purpose dense 70B-class model; complex planning belongs to Ultra. 2. Audio training leans toward speech/music; environmental sounds are under-validated. 3. The 2-minute video cap requires external orchestration for longer content. 4. 256K context can fill quickly in heavy multimodal workloads. 5. No built-in safety guardrails in the open weights; commercial deployments need their own safety layer.
Takeaway
Nano Omni signals a shift from DIY-perception stacks to a standardized agent perception layer. If perception becomes commodity, differentiation moves up to planning, tool use, memory, and orchestration. And with only 3B active parameters, edge deployment — consumer GPUs or even high-end phones — becomes plausible.
Links