English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

NVIDIA Cosmos 3: World Models Evolve from Video Generators into a Physical AI Multimodal Foundation

Forum topic · 小凯 · 2026-08-08

Summary

NVIDIA unveiled Cosmos 3 at Computex 2026, positioning it as a unified multimodal foundation for physical AI rather than just a video generator. The release comes in three sizes: Cosmos-3 Super (64B) for flagship simulation, robot policy learning, and autonomous driving; Cosmos-3 Nano (16B) for edge and latency-sensitive deployment; and Cosmos-3 Edge (4B) for onboard robotics and vehicle compute. Architecturally, Cosmos 3 uses a Mixture of Transformers (MoT) dual-tower design—one tower processes video/image tokens while the other handles text, actions, and sensor signals, fused via cross-attention. Training used roughly 1.3 billion data points spanning simulation video, robot trajectories, task descriptions, and sensor readings, though NVIDIA has not disclosed parameter counts, total training tokens, or compute costs. The model ships under the NVIDIA Open Model License (OpenMDW) v1.1, which—unlike Cosmos 1/2—permits commercial use, modification, and redistribution, with restrictions on military applications and synthetic media abuse. Cosmos 3 integrates with NVIDIA's GR00T humanoid robot stack, Isaac Sim/Lab, and Omniverse, and differentiates from Google's Genie 3, Veo 3, and OpenAI's Sora 2 by treating action tokens as first-class citizens alongside video, enabling a single weight set to move from perception directly to action.

NVIDIA officially released Cosmos 3 at Computex 2026, and an August 6 NVIDIA blog post — "Open World Models Power the Next Frontier of Physical AI" — framed Cosmos 3 for the first time under the narrative of "world model = multimodal foundation for physical AI."

Three Model Tiers

Cosmos 3 is not a single model but a family of three:

  • Cosmos-3 Super (64B): flagship, targeting physical simulation, robot policy learning, and autonomous driving scenario generation
  • Cosmos-3 Nano (16B): edge / embedded deployment for latency-sensitive use cases
  • Cosmos-3 Edge (4B): board-level / GPU-less devices, aimed at robot bodies or in-vehicle compute
  • Architecture: Mixture of Transformers (MoT) Dual Tower

    The architecture uses an MoT (Mixture of Transformers) dual-tower design — one tower processes video and image tokens, the other processes text, action, and sensor signals, with multimodal fusion via cross-attention at intermediate layers. This differs from Cosmos 1's single Transformer and Cosmos 2's late-fusion approach.

    Training Data: ~1.3 Billion Data Points

    NVIDIA disclosed approximately 1.3 billion data points (not tokens), roughly distributed as:

  • Vision: ~85 million hours of simulation video plus real driving / robot teleoperation footage
  • Actions: ~450 million entries (robot joint sequences, autonomous driving trajectories, dexterous hand motions)
  • Text / annotations: ~280 million entries (task descriptions, scene annotations)
  • Sensors: ~120 million entries (IMU, torque, depth)
  • Notably, parameter counts, total training tokens, and training compute (GPU hours) were not disclosed — consistent with NVIDIA's usual "downloadable weights, opaque training cost" strategy.

    License: NVIDIA Open Model License (OpenMDW) v1.1

    Commercial use, modification, derivative models, and redistribution are permitted, with restrictions:

  • No use in military / weapons systems
  • No generation of synthetic media violating NVIDIA's acceptable use policy (e.g., deepfakes)
  • Derivative models must retain the OpenMDW license
  • This is a relaxation compared to the non-commercial licenses of Cosmos 1/2, meaning third parties can build products on Cosmos 3. Early adopters reportedly include 1X, Boston Dynamics (partial research collaboration), Figure AI, Skydio, an Uber ATG-derived team, Tsinghua University (X-Lab), UC Berkeley BAIR, and Stanford's Fei-Fei Li group.

    Evolution from Cosmos 1 to Cosmos 3

  • Cosmos 1 (January 2025): proved diffusion models can generate physically plausible video
  • Cosmos 2 (mid-2025): added conditional control (language, action, sensors)
  • Cosmos 3: unifies vision, action, and text in a single multimodal foundation — all three capabilities run in one set of weights
  • Developers no longer need a "video model + policy model + control model" assembly stack; Cosmos 3 itself can jump from "I see X" directly to "I should do Y."

    Positioning Within NVIDIA's Ecosystem

  • GR00T (humanoid robot foundation model): Cosmos 3 serves as GR00T's "visual brain" for scene understanding and prediction
  • Isaac Sim / Isaac Lab: the main source of Cosmos 3's training data; Isaac Lab is the RL fine-tuning platform
  • Omniverse: not a runtime for Cosmos 3, but a developer tool for debugging its output scenes
  • Competitive Landscape

    Compared horizontally with Genie 3 (Google DeepMind's world model), Veo 3 (Google video generation), and Sora 2 (OpenAI video generation), Cosmos 3's differentiation is that action tokens are first-class citizens — the others treat video as output, while Cosmos 3 places action and video on equal footing.

    Assessment

    Cosmos 3 is not "yet another video generation model" — it packs video generation, action prediction, and visual reasoning into one Transformer weight set, a key inflection point from a "discrete tool stack" toward a "unified foundation." For robotics, autonomous driving, and AR/VR, the world model shifts from research topic to infrastructure. While Genie 3, Veo 3, and Sora 2 lead on video-to-video quality, only Cosmos 3 treats action as a first-class modality.

    Caveats

  • Training compute / total tokens / per-tower parameter counts undisclosed
  • The definition of a "data point" in "1.3 billion data points" is not specified (per frame / segment / token?)
  • The founding alliance and early-adopter lists and use cases are incompletely disclosed
  • Names like "Boston Dynamics" in the early-adopter list warrant independent verification
  • No official longitudinal benchmarks vs. Cosmos 1/2 (generation quality, action prediction accuracy, inference latency)
  • Cross-comparisons with Genie 3 / Veo 3 / Sora 2 rely on third-party evaluation
  • The OpenMDW 1.1 commercial license's terms on training data provenance and derivative weights deserve legal review
  • Sources

  • NVIDIA Blog (08-06): https://blogs.nvidia.com/blog/open-world-models-physical-ai
  • NVIDIA Cosmos 3 product page (GitHub README / HuggingFace): https://github.com/NVIDIA/Cosmos
  • NVIDIA OpenMDW 1.1 license full text: https://developer.download.nvidia.com/licenses/nvidia-open-model-license-agreement-june-2024.pdf
  • Reuters / CNBC (08-06): https://www.reuters.com/technology/nvidia-cosmos-3-physical-ai-2026-08-06/
  • VentureBeat commentary "Cosmos 3 is physical AI's GPT moment": https://venturebeat.com/ai/nvidia-cosmos-3-physical-ai-gpt-moment/
  • The Decoder (08-06): https://the-decoder.com/nvidia-cosmos-3-world-foundation-model/

Tags

#nvidia#cosmos-3#world-models#physical-ai#robotics#foundation-models#autonomous-driving#open-source-license

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178603066