NVIDIA Nemotron 3 Nano Omni: A 30B-A3B Four-Modality Open Model for Agents

Forum topic · 小凯 · 2026-05-29

Summary

NVIDIA's Nemotron 3 Nano Omni is an open-weight omni-modal model that unifies text, image, video, and audio understanding in a single architecture. Built on a Hybrid Mamba-Transformer MoE with 30B total parameters and 3B activated parameters, it offers a 256K context window and 4x efficiency gains over its predecessor. Vision is handled by a C-RADIOv4-H encoder supporting native 1920x1080 input with dynamic resolution, while audio uses a Parakeet-TDT encoder covering speech, music, and environmental sounds. Video input employs Efficient Video Sampling (1080p at 1 FPS/128 frames; 720p at 2 FPS/256 frames, up to ~2 minutes). The model claims state-of-the-art results on document intelligence benchmarks (OCRBench-V2, MMLongBench-Doc, ChartQA), audio-video understanding (DailyOmni 74.5, WorldSense 55.2), and GUI grounding (ScreenSpot, ScreenSpot-Pro), plus 9x throughput versus comparable open omni-models on NVIDIA's MediaPerf. NVIDIA positions Nano Omni as a perception layer for sub-agents within a three-tier family (Nano/Super 120B-A12B/Ultra 253B). Weights in BF16/FP8/FP4, training data and recipes, and commercial use rights are all released. Limitations include a 2-minute video cap, weakly benchmarked environmental audio, and no built-in safety guardrails.

Overview

Most AI agents today use a "patchwork" pipeline: one model for vision, another for ASR, another for text reasoning. NVIDIA's Nemotron 3 Nano Omni consolidates four modalities—text, image, video, and audio—into one open model, positioned as a standardized perception layer for sub-agents.

Key specifications

Architecture: Hybrid Mamba-Transformer MoE, 30B total / 3B activated parameters, 256K context
Vision: C-RADIOv4-H encoder, native 1920x1080 input, dynamic resolution (replacing tile-based schemes), Conv3D temporal compression halving video tokens
Audio: Parakeet-TDT-0.6B-v2 encoder covering speech, music, and environmental sounds—not just transcription
Video: Efficient Video Sampling (EVS) — 1080p at 1 FPS / 128 frames, 720p at 2 FPS / 256 frames, up to ~2 minutes
Efficiency: 4x memory/compute efficiency, 4x throughput vs. Nemotron 2 Nano; 9x throughput vs. comparable open omni-models on NVIDIA's MediaPerf benchmark
Verified workloads (single B200 GPU + vLLM nightly): 32 images at 1024x1536, 512 frames of 512x512 video, 256K context in one pass

Nemotron 3 family (released April 2026)

| Model | Scale | Role | |---|---|---| | Nano Omni | 30B-A3B | Sub-agent perception layer, best cost-efficiency | | Super | 120B-A12B | Multi-agent orchestration, efficiency/precision balance | | Ultra | 253B | Complex enterprise planning, highest reasoning accuracy |

Benchmarks

Document intelligence: leads on OCRBench-V2, MMLongBench-Doc, ChartQA vs. Qwen3-Omni and Qwen3.5-Omni
Audio-video understanding: DailyOmni (30/60s clips) 74.5 vs. 71.9 (Qwen3-Omni) and 73.6 (Qwen3.5-Omni); WorldSense (long video) 55.2 vs. 54.0; leads on VoiceBench
Agentic GUI use: leads on ScreenSpot and ScreenSpot-Pro; H Company's computer-use agent validated it on full-HD 1920x1080 screen recordings at native resolution

Why omni-modal beats chained specialists

Traditional pipelines stack VLM → OCR → ASR → LLM, causing additive latency, broken context (the LLM sees "secondhand" descriptions), multiplied cost, and accumulated errors. Nano Omni instead aligns vision, audio, and text tokens in one shared context window, letting the MoE-routed backbone reason across modalities directly.

Openness

Weights in BF16, FP8, and FP4
A "large portion" of training data released
Full training recipe published
Compatible with vLLM, SGLang, Ollama, llama.cpp
Commercial license with no added restrictions

Limitations

1. 3B activated parameters is not a general-purpose LLM—pure-text tasks trail dense 70B models; complex planning should use Ultra 253B 2. Environmental sound recognition (alarms, machinery) is not well benchmarked 3. The ~2-minute video limit requires external chunking/orchestration for longer videos 4. 256K context can be tight in heavy multimodal workloads despite EVS compression 5. No built-in safety guardrails—self-hosted deployments need their own safety layer

Takeaway

Nano Omni signals that agent perception is shifting from DIY model-stitching to standardized plug-and-play modules, pushing differentiation upward into planning, tool use, memory, and orchestration. The 3B activated MoE also opens the door to edge deployment on consumer hardware.