Glonce

NVIDIA adds four open Cosmos 3 Nano checkpoints for world simulation

modelsThe Glonce editorsHugging Face🔥71Glonce rating71Impact65Novelty60Relevance100Surprise35Players90Credibility85
NVIDIA adds four open Cosmos 3 Nano checkpoints for world simulation

NVIDIA's Cosmos world model development platform has added four open-weight checkpoints to the Cosmos 3 model family, announced in a Hugging Face blog post. They target interactive world simulation and multi-sensor, multi-view driving scenes. All four are post-trained examples of Cosmos 3 Nano and have 16B trainable parameters each. They use the OpenMDW1.1 license, which permits commercial and non-commercial use under its terms. A technical report, demos, checkpoints and notebooks are linked from the post.

Cosmos3-Nano-Transfer-Auto is a bidirectional model for offline synthetic driving data generation. It produces synchronized video across up to 11 camera views, including fisheyes, and supports joint RGB and LiDAR generation. Per-camera World Scenario Map (WSM) controls specify scene structure and traffic motion, while text prompts guide appearance. Developers can vary weather, lighting and time of day to widen training and evaluation coverage, optionally anchoring appearance with initial RGB images. Joint LiDAR generation also needs aligned HD-map range-map controls. Transfer-Auto is also meant as the starting point for adapting Cosmos to your own sensor rig and driving domain: post-train it on your data, then use it for offline generation or put it through causal training and few-step distillation for streaming.

Cosmos3-Nano-Sim-Auto is that streaming result: the causally distilled variant of Transfer-Auto. It generates successive small chunks of synchronized camera video and LiDAR in just a few denoising steps, using generated history and updated driving controls for long, interactive rollouts. Inputs are per-camera captions and synchronized WSM controls, optionally initial RGB images, plus HD-map range controls aligned with the cameras for LiDAR. Closed-loop use requires a simulation runtime and a WSM renderer to update the controls from driving inputs.

Cosmos3-Nano-Sim-Transfer sits on top of a physics simulator. The simulator computes motion, contacts and physical state; Sim-Transfer takes its spatial controls (depth, segmentation, edges, blur) and a text prompt and renders color observations that a policy can consume. The loop is: action, simulator update, spatial controls, generated RGB, next policy action. Text guides appearance, while physics and ground-truth state stay with the simulator.

Cosmos3-Nano-Sim-Bimanual works differently: it generates the next observation directly from the scene and the robot's actions, with no simulator in the loop. It supports interactive two-arm simulation for YAM and AgiBot embodiments, controlled by a policy or human-operated joysticks. A user starts with an image, moves the robot or camera, and generates the next observations; one control sequence can switch between viewpoint changes and manipulation. The mixed-action example uses 59 control values, padded to 64 during training. The schema defines coordinate frames, rotations, gripper calibration, validity masks and control domains. Timing must be aligned so that T transitions have T+1 observations, and released inference actions are already normalized.

On the mechanics, Cosmos 3's Mixture-of-Transformers backbone connects separate text-processing and visual-generation networks through attention. The generator predicts each short segment from the prompt, available controls and generated history. It works on compressed video latents, where after the initial image each latent time step represents four RGB frames. Transfer predicts one step per chunk; the optimized Bimanual setup in the report predicts two. Temporal-causal attention lets the model read history and controls while blocking future target observations, and a key/value cache carries attention information forward.

The training recipe has four stages. Stage 0, bidirectional adaptation, learns the domain and control-to-video relationships from complete recorded clips. Stage 1, teacher forcing, predicts each chunk from earlier recorded observations and controls with the clean answer and future targets masked. Stage 2, consistency distillation, uses a frozen teacher to train a student that generates each chunk in four denoising steps. Stage 3, self-forcing, trains on the student's own generated history, with a frozen teacher and an auxiliary score model supplying the distribution-matching signal. NVIDIA says the last stage matters because a slightly mispredicted object becomes context for later frames, so errors compound.

The post-training recipes start from Cosmos3-Nano or compatible base weights and cover all four stages. For running inference, the requirements are Linux, an NVIDIA GPU, Docker and NVIDIA Container Toolkit. The published vLLM-Omni setup was tested on one GB200, which the post says is not a minimum-memory specification. NVIDIA frames the release as an open foundation for researchers and developers, and adds that these problems remain hard: the work ahead is to measure where the models faithfully support policy learning and evaluation, understand their limits, and improve where they fall short.

Key facts

  • NVIDIA added four open-weight checkpoints to Cosmos 3: Cosmos3-Nano-Transfer-Auto, -Sim-Auto, -Sim-Transfer and -Sim-Bimanual. Each has 16B trainable parameters and is a post-trained example of Cosmos 3 Nano.
  • Transfer-Auto generates synchronized video across up to 11 camera views, including fisheyes, with joint RGB and LiDAR generation; Sim-Auto is its causally distilled streaming variant.
  • Sim-Transfer renders RGB from a physics simulator's depth, segmentation, edge and blur controls plus text; Sim-Bimanual predicts video directly from robot or camera actions for YAM and AgiBot embodiments.
  • A four-stage recipe (bidirectional adaptation, teacher forcing, consistency distillation, self-forcing) yields a student that generates each chunk in four denoising steps.
  • The models use the OpenMDW1.1 license, allowing commercial and non-commercial use under its terms; inference needs Linux, an NVIDIA GPU, Docker and NVIDIA Container Toolkit.

Why it matters

World models that respond to actions are a building block for testing driving and robot policies without a real vehicle or arm. This release covers two ways to get there. Sim-Transfer keeps a physics simulator in charge of motion and state and uses the model only to render realistic visuals. Sim-Bimanual drops the simulator and predicts the next observation straight from the scene and the robot's actions. For driving, the checkpoints generate camera video and LiDAR together, so appearance and 3D geometry agree. NVIDIA also publishes post-training recipes for all four stages, which makes the checkpoints a starting point for adaptation rather than a fixed product.

Who it affects

Autonomous-driving teams that want synthetic multi-camera and LiDAR data across weather, lighting and time of day, or a streaming simulator for their own sensor rig. Robotics researchers working on two-arm manipulation, especially on YAM and AgiBot embodiments. World-model researchers who want open weights and recipes to compare methods and contribute improvements, which is how NVIDIA describes its aim.

How to use it

The checkpoints are downloadable from Hugging Face, and the post links a technical report, demos and notebooks for each model; it also points to open skills that a coding agent can follow to run inference. Running inference requires Linux, an NVIDIA GPU, Docker and NVIDIA Container Toolkit, and the models use OpenMDW1.1, which permits commercial and non-commercial use under its terms. For a new sensor rig or domain, adapt Transfer-Auto first, then distill it into Sim-Auto. For other setups, the recipes cover data preparation or validation, stage configurations, checkpoint handoffs and export. The post advises validating a sample and testing training and restart before a longer run, and keeping full training checkpoints, since exported inference weights omit optimizer and other training state.

How solid is it

The source is NVIDIA's own announcement on the Hugging Face blog, so everything rests on the vendor's description. It is specific about architecture, inputs, the training recipe and setup requirements, and links a technical report. No benchmark results, quality metrics or comparisons with other world models are reported in the text, and no inference speed, latency or frame-rate figures are given for streaming generation.

Risks and caveats

NVIDIA itself says these problems remain hard and that the work ahead is to measure where the models faithfully support policy learning and evaluation, understand their limits, and improve where they fall short. Closed-loop use of Sim-Auto needs a simulation runtime and WSM renderer, and LiDAR generation needs aligned HD-map range controls. Bimanual users must follow the action schema and timing rules or the controls will not describe the intended motion. The published vLLM-Omni setup was tested on one GB200, and the post says this is not a minimum-memory specification, so the hardware needed for smaller GPUs is not stated. A short successful training run verifies execution, not convergence, so quality should be checked on held-out episodes.

“a successful short run verifies execution, not convergence”

— NVIDIA, Hugging Face blog post on Cosmos 3 interactive world simulation