4B World Action Model Runs at 15Hz on Jetson: NVIDIA Cosmos 3 Edge Brings Robot Inference to the Edge

Forum topic · 小凯 · 2026-07-21

Summary

NVIDIA has released Cosmos 3 Edge, a 4-billion-parameter world action model designed for robots, smart cameras, and edge devices. The model pairs an autoregressive Transformer for vision-language understanding with a diffusion Transformer for visual, audio, and action token generation, sharing multimodal attention layers while keeping separate normalization and MLP blocks. Actions are encoded as geometric vectors covering translation, rotation, and manipulation states, enabling both forward prediction (given an action, predict resulting visuals) and inverse inference (given visual changes, infer the causing action). NVIDIA reports 15Hz control on Jetson Thor by generating 32 actions per inference at 640x360, plus a DROID post-trained policy for grasping and placement. The model card notes caveats: no explicit physics simulator, state drift on long-horizon tasks, and implausible outputs in out-of-distribution settings. Licensed under OpenMDW 1.1, tested only on Linux with BF16.

4B World Action Model Runs on Jetson: NVIDIA Cosmos 3 Edge Packs Perception and Action into One Edge Model

Source: NVIDIA / Hugging Face Date: 2026-07-20

Release post: https://huggingface.co/blog/nvidia/cosmos3edge
Model card: https://huggingface.co/nvidia/Cosmos3-Edge
Model collection: https://huggingface.co/collections/nvidia/cosmos3

Yesterday the Cosmos conversation was about a robotics alliance of 22 Japanese companies. Today NVIDIA has released the actual model that alliance plans to use: Cosmos 3 Edge, 4 billion parameters, aimed at robots, smart cameras, and edge devices.

Its goal goes beyond recognizing scenes. Given the current visual state and a task, the model can produce reasoning text, generate action tokens, and predict the visual consequences of those actions. NVIDIA calls it a world action model. In plain terms, the model must both answer "what is happening in front of me" and "what should I do next, and what will happen after I do it."

Two Transformers, One Shared World Representation

Cosmos 3 Edge is not a shrunken VLM. It has two Transformers:

An autoregressive tower handles vision and text tokens, responsible for understanding, reasoning, and text output.
A diffusion tower handles visual, audio, and action tokens, responsible for prediction, generation, and neural simulation.

Each tower keeps its own normalization layers and MLPs, but the two share multimodal attention layers. Actions are encoded as geometric vectors covering translation, rotation, and manipulation states. Vehicle ego-pose, camera motion, robotic arm end-effector pose, and gripper open/close all map into the same action representation.

A practical benefit of this design: the same model can do forward prediction — give an action, see the future frames — and inverse inference — given a change, guess the action that caused it. Policy mode ties the two together, generating expected visual consequences alongside the actions it outputs.

NVIDIA also released Cosmos3-Edge-Policy-DROID, post-trained on DROID robot data for grasping and placement tasks, with the accompanying training scripts public. Developers can fine-tune on a small H100 cluster or DGX Station, then deploy down to Jetson, GeForce RTX, RTX PRO, or data center GPUs.

Attractive Edge Numbers — But Check the Fine Print

The headline figures from the release post: 640×360 robot control observations, 32 actions generated per inference on Jetson Thor, achieving a 15 Hz control frequency. NVIDIA also claims first place on VANTAGE-Bench visual analysis among same-size 4B models and SOTA on robot policy learning.

The model card's detailed tables are more complicated. It lists Jetson AGX Thor T5000, T3000, T2000, H100, B200, RTX PRO 6000 and other hardware; Reasoner first-token latency on RTX PRO 6000 at single concurrency is 239ms, with a request latency of roughly 639ms for 100 output tokens. Meanwhile, the DROID Policy generation latency table shows 6.32 seconds on Jetson AGX Thor T5000.

The apparent conflict between 15 Hz and 6.32 seconds likely reflects different inference backends, action chunking, and timing conventions: the former emphasizes maintaining control frequency after one inference chunk produces 32 actions, while the latter is a full Policy generation call. NVIDIA did not clarify this relationship in the release post. Anyone actually deploying on a robot should not cherry-pick the 15 Hz figure for marketing — end-to-end closed-loop testing on target hardware is essential.

The license is OpenMDW 1.1, usable for commercial and non-commercial scenarios, but it is not Apache-2.0. The runtime environment has only been officially tested on Linux with BF16; Windows, FP4, FP8, and FP16 are outside the official guarantee.

Why This Approach Is Worth Following

MiniCPM-Robot, LingBot-VA, and Cosmos 3 Edge represent three distinct strategies. MiniCPM-Robot bets on small models and streaming memory; LingBot bets on complete embodied infrastructure; NVIDIA bets on a vertical closed loop of "world model + action model + Jetson hardware."

NVIDIA's advantage is concrete. A robotics team receives not just weights but an entire chain: training framework, post-training scripts, edge inference, robot reference policies, and GPU/Jetson deployment platforms. Even if the model's raw performance leads only slightly, the engineering closed loop may determine who first enters factories and warehouses.

The Model Card Pours Cold Water on Itself

But don't write this off as "general-purpose robotics is solved." The model card is explicit: Cosmos 3 has no explicit physics simulator; it only approximates 3D geometry, contact dynamics, and object permanence from data. Long-horizon tasks will show state drift, out-of-distribution environments may produce implausible actions, and objects may vanish, deform, or clip through each other. NVIDIA states clearly that outputs must not be treated as physically accurate simulation or used directly for safety-certified decisions.

What matters most next is real-robot data: how many new trajectories cross-robot transfer requires, whether 15 Hz holds up under occlusion, deformable objects, and contact-rich tasks, and whether there is a viable deployment path on non-NVIDIA hardware. Cosmos 3 Edge has already compressed world models from the data center down to the edge. The next hurdle isn't producing another polished video — it's making actions run continuously on real machines for hours without failure.