NVIDIA Cosmos 3: The First Open Physical-AI Omnimodel

NVIDIA Cosmos 3 is the first fully open physical-AI omnimodel — a single model that natively understands and generates text, images, video and ambient sound, and also outputs the action signals a robot needs to move. Announced at GTC Taipei / Computex on May 31, 2026, its weights are downloadable from Hugging Face today.

Earlier physical-AI systems chained separate specialist models — one for scene understanding, another for world generation, a third for predicting actions — and paid an integration tax at every handoff. Cosmos 3 collapses that pipeline into one Mixture-of-Transformers architecture, which is the part of this release that actually changes how robotics and autonomous-vehicle teams build.

This guide covers what shipped, what "omnimodel" means in practice, the two-tower design, the five use modes a single model supports, the three hardware tiers, NVIDIA's leaderboard claims (and why we read every one of them as vendor-stated), and how the OpenMDW-1.1 license changes the calculus for commercial products.

NVIDIA announced Cosmos 3 during Jensen Huang's GTC Taipei keynote at Computex 2026 on May 31, 2026, positioning it as the open frontier foundation model for physical AI. The launch arrived in the same event cycle that NVIDIA used to frame the rest of its physical-AI roadmap — for the wider keynote context, see our GTC Taipei / Computex 2026 keynote first take and the broader order-pipeline analysis from the same keynote.

Cosmos 3 is not NVIDIA's first move into world models. The original Cosmos family, released in 2025, shipped separate models for world generation, physical understanding, and controlled scene generation. Cosmos 3 is best understood as a unification: it folds what previously required chaining several specialist models into one architecture that runs in a single forward pass. Two of the three tiers — Nano and Super — are downloadable now; the smaller Edge tier is announced but not yet released.

NVIDIA's term for Cosmos 3 is "omnimodel": one model that natively understands and generates text, images, video, and ambient sound, and additionally produces robot action signals. The label matters less than what it removes. Modality encoders — a Vision Transformer for vision, a Variational Autoencoder for generation, and domain-aware vectors for actions — all project into a shared representation space, so the model reasons across modalities instead of bolting them together with glue code.

The differentiator over prior generative world models is the action output. Cosmos 3 does not just describe or render a scene; it can emit numerical control signals — joint angles, gripper positions, and trajectory points — that a robot controller or AV stack can execute directly. That is the difference between a model that imagines the world and one that can be wired into a physical control loop.

Here is why the pipeline-collapse framing is the right lens. Before an omnimodel, a robotics team typically ran a perception model to understand a scene, a separate world model to simulate outcomes, and a third policy model to predict actions — passing outputs between them and absorbing integration error and latency at every boundary. A single model that handles all of those tasks does more than improve any one score: it removes inference steps, cuts handoff latency, and simplifies the MLOps stack a team has to maintain. NVIDIA frames this as compressing physical-AI training and evaluation cycles "from months to days" — a vendor claim worth testing against your own workload rather than taking at face value.

Cosmos 3 is built as a Mixture-of-Transformers (MoT) with a two-tower design. The two towers split the work — and the dependency direction between them is the whole point.

Reasoner Tower — autoregressive vision-language model

The Reasoner is an autoregressive vision-language model that ingests the scene and the instruction and builds a structured understanding of what is happening and what should happen next. It is the "think before you act" half of the model: it produces the context that conditions everything the Generator does.

Generator Tower — diffusion transformer

The Generator is a diffusion-based transformer that produces the output — future video frames, generated worlds, or action trajectories. Critically, the Generator cannot run without the Reasoner's context. Generation is always conditioned on reasoning, rather than the two operating as independent stages you could swap out.

Shared spatial-temporal structure

Both towers share a 3D multi-dimensional rotary position embedding (mRoPE), which gives them a consistent sense of spatial and temporal structure across modalities. That shared coordinate system is part of what lets reasoning and generation stay aligned on the same scene rather than drifting apart.

The same Cosmos 3 weights support five distinct use modes, which is the practical expression of the omnimodel claim. Rather than picking a specialist model per task, a builder selects a mode based on the input and output they need. The capability map below pairs each mode with a concrete robotics or autonomous-vehicle example.

Most launch coverage skips the hardware reality, but the tier you choose is dictated by where you deploy. The decision matrix below maps each tier to its hardware target, primary use case, and availability — including the Edge tier that is announced but not yet downloadable, so device-side teams know not to plan around it arriving today.

NVIDIA says Cosmos 3 ranks #1 across a wide set of physical-AI leaderboards. Every one of these is stated among open models, and we treat them all as vendor-stated: NVIDIA had not published point-score comparison tables at launch, and independent reproduction of the rankings was not available. These leaderboards are also new, with limited third-party audit history. We list the claims qualitatively below — no invented scores — because the honest version is more useful than a precise number we cannot verify.

What can be said with confidence: the rankings span three distinct problem families — world generation, robot policy, and vision understanding — which is itself notable for a single model, since most systems specialize. What cannot be said yet: how Cosmos 3 compares to closed frontier physical-AI systems, or how the open-model rankings hold up under independent evaluation. For sophisticated teams, the right move is the same one we recommend for any new model — run the eval on your own scenes and tasks, not on the press release.

Cosmos 3 ships under OpenMDW-1.1, a license stewarded by the Linux Foundation. It permits commercial use, training, modification, redistribution, and derivative models. The one notable constraint: products built on Cosmos must display "Built on NVIDIA Cosmos" somewhere visible — a website, UI, about page, or documentation. NVIDIA does not claim ownership of outputs generated from Cosmos or its derivatives.

The licensing story is bigger than one model. NVIDIA adopted OpenMDW-1.1 across four model families at once — Cosmos, Isaac GR00T, Ising, and Nemotron — standardizing open-model licensing across its physical-AI, robotics, quantum, and general foundation-model lines. For the coding-model side of that same strategy, see our coverage of NVIDIA's Nemotron open-model strategy. Note that OpenMDW-1.1 is distinct from NVIDIA's prior NVIDIA Open Model License — Cosmos 3 uses the Linux Foundation standard.

A full builder toolchain ships with the model: Cosmos Curator for data filtering, annotation, and deduplication; Cosmos Evaluator for output scoring; NVIDIA TAO 7 for fine-tuning; and the Cosmos Cookbook of domain-specific recipes, with post-training scripts on the github.com/nvidia/Cosmos repository. If you are weighing an open physical-AI model against alternatives for a specific pipeline, our AI & digital transformation engagements start with exactly this kind of comparative, scenario-grounded evaluation.

Cosmos 3 also launched with a coalition and a roster of early adopters, which together signal where this is headed. The NVIDIA Cosmos Coalition has six founding members — Agile Robots, Black Forest Labs, Generalist, LTX, Runway, and Skild AI — drawn deliberately from different corners of the field: Agile Robots is a hardware manufacturer, Black Forest Labs, LTX, and Runway are generative-video labs, Skild AI works on generalist robot policy, and Generalist rounds out the policy side. The coalition's premise is a shared ecosystem with DGX Cloud infrastructure access and open contribution — a different model from a typical vendor-partner program.

Agile Robots is already an early-access partner, using Cosmos 3 to generate action-conditioned training trajectories for its Thor 3 and FR3 robots on complex industrial pick-and-place tasks, with testing inside the European Industrial AI Cloud. The launch adopter list also spans Samsung, LG Electronics, and Doosan Robotics in robotics; Li Auto in autonomous vehicles; and Centific, Fogsphere, Linker Vision, Milestone Systems, and Yuan in vision AI.

For builders, the decision tree depends on what you are building. The matrix below sorts the most common physical-AI workload classes — and for the broader open-model landscape this fits into, our open-weight frontier models retrospective and our humanoid robotics pipeline coverage give useful context.