NVIDIA/cosmos

Cosmos

Website | Framework | Agent Skills | Models

Table of Contents

• Introduction
• Cosmos 3
• Key Capabilities
• Model Architecture
• Model Family
• Supported Generation Settings
• Input and Output
• Use Cases
• Generator
• Reasoner
• Quickstart
• Generator with Diffusers
• Generator with vLLM-Omni
• Generator with NIM
• Generator with SGLang
• Reasoner with Transformers
• Reasoner with vLLM
• Reasoner with TensorRT-LLM
• Reasoner with NIM
• Troubleshooting
• Which CUDA version should I use?
• Which base container should I use?
• torch.cuda.is_available() is False
• Import fails with libxcb.so.1: cannot open shared object file
• uv errors on install or sync
• Choosing an Integration
• Examples
• Inference Benchmarks
• Finetune
• Export and Convert Checkpoints
• Distill
• Limitations
• Ecosystem
• News
• License and Contact

Introduction

NVIDIA Cosmos is an open platform of world models, datasets, and tools that enables developers to build Physical AI for robots, autonomous vehicles, smart infrastructure, and more.

Cosmos 3

Cosmos 3 is our newest model family [[Models]](https://huggingface.co/collections/nvidia/cosmos3) [[Report]](https://research.nvidia.com/labs/cosmos-lab/cosmos3/technical-report.pdf) [[Website]](https://research.nvidia.com/labs/cosmos-lab/cosmos3/). It is a suite of omnimodal world models designed to jointly process and generate language, images, video, audio, and action sequences within a unified Mixture-of-Transformers architecture. By supporting highly flexible input-output configurations, it seamlessly unifies critical modalities for Physical AI — effectively subsuming vision-language models, video generators, world simulators, and world-action models into a single framework.

Cosmos 3 exposes two runtime surfaces:

| Surface | Inputs | Outputs | Use Cases | |----------|----------|----------|----------| | Reasoner | Text, vision | Text | World understanding, grounding, physical reasoning, task planning, action forecasting, embodied agent reasoning, and autonomous system decision making | | Generator | Text, vision, sound, action | Vision, sound, action | World generation, world simulation, future prediction, synthetic data generation, policy learning, and robot training |

Key Capabilities

• World understanding: Analyze videos and images for captions, temporal events, next actions, spatial grounding, physical plausibility, and causal outcomes.
• World generation: Produce images, videos, synchronized sound, and action-conditioned rollouts from text, image, video, or action inputs.
• Action modeling: Predict policy actions, inverse dynamics, and forward dynamics for robotics, camera motion, egocentric motion, and autonomous-driving settings.
• Research and production paths: Use Diffusers and Transformers for Python-first development, vLLM-Omni, vLLM, TensorRT-LLM, or SGLang for OpenAI-compatible serving, and NIM containers for turnkey Reasoner serving or Generator deployment for text-to-video and image-to-video generation.
• Post-training recipes: Adapt vision, action, and reasoner workflows with Cosmos Framework training recipes and task-specific evaluation [Coming Soon].

Model Architecture

Cosmos 3 is an omnimodal world model built on a unified Mixture-of-Transformers (MoT) architecture that combines an autoregressive (AR) transformer for reasoning with a diffusion transformer (DM) for multimodal generation. In Reasoner Mode, language and visual understanding tokens are processed through causal self-attention, enabling next-token prediction for tasks such as perception, planning, and world reasoning. In Generator Mode, noisy image, video, audio, and action tokens are denoised through full attention, allowing the model to jointly generate coherent multimodal outputs. Both modes share the same transformer architecture, multimodal attention layers, and a unified 3D multi-dimensional rotary position embedding (mRoPE) representation that encodes spatial and temporal structure across modalities, enabling consistent reasoning over images, videos, audio streams, and action trajectories.

Model Family

¹ The models generate sound along with the video, not standalone.
² Cosmos3-Edge currently doesn't support video-to-video transfer.

Supported Generation Settings

| Setting | Supported values | | ------------------| --------------------------------------- | | Resolution tiers | 256p, 480p, 720p, default=480p | | Aspect ratios | 16:9, 4:3, 1:1, 3:4, 9:16, default=16:9 | | Frame rates | 10, 16, 24, and 30 FPS, default=24 | | Frame count | 5 to 300 frames, default=189 | | Precision | BF16 tested | | Operating system | Linux | | GPU architectures | NVIDIA Ampere, Hopper, and Blackwell |

Cosmos3-Edge only supports 256p and 480p resolution, 12–30 fps, and 50–150 frames.

Input and Output

| Spec | Value | | --- | --- | | Input types | Text, text + image, text + video, text + image + action | | Input formats | Text string, JPG/PNG/JPEG/WEBP image, MP4 video, JSON action array | | Vision conditioning | 720p uses 1280x720, 480p uses 832x480, and 256p uses 320x192. Video conditioning uses 5 frames at the matching resolution. | | Action conditioning | Supported action dimensions depend on the embodiment, including camera motion (9D), autonomous vehicle (9D), egocentric motion (57D), single-arm robot (10D, DROID/UR/Fractal/Bridge/UMI), dual-arm robot (20D, dual DROID arms), humanoid robot (29D, AgiBot). | | Output types | Image, video, sound, action state, text | | Output formats | JPG image, MP4 video, AAC sound stream muxed into MP4, JSON action values, text string | | Prompt length | Fewer than 300 words is recommended for world-generation prompts | | Sound output | Stereo AAC at 48 kHz when generated with video |

Use Cases

Generator

Generator examples produce non-text outputs conditioned by text, vision, and action inputs.

| Workflow | Inputs | Outputs | What it demonstrates | | --- | --- | --- | --- | | Text-to-image | Text | Vision | Robotics laboratory scene generation from a text prompt | | Text-to-video | Text | Vision | Industrial video generation from a dense scene description | | Text-to-video with sound | Text | Vision, sound | Synchronized visual and audio generation | | Image-to-video | Text, image | Vision | Robot manipulation animation from a starting image and prompt | | Image-to-video with sound | Text, image | Vision, sound | Image-conditioned motion with synchronized audio | | Video-to-video | Text, video | Vision | Prompt-guided transformation of a robot manipulation video | | Video-to-video with sound | Text, video, sound | Vision, sound | Prompt-guided transformation of a robot manipulation video | | Forward dynamics | Text, vision, action | Vision | Future-state rollout from action and visual context | | Action policy | Text, vision | Action, vision | Action trajectories and rollout video from context |

Generator prompt upsampling expands short scene descriptions into dense structured prompts. The current examples use these sampling defaults:

| Parameter | Value | | --- | ---: | | max_tokens | 20000 | | temperature | 0.7 | | top_p | 0.8 | | top_k | 20 | | repetition_penalty | 1.0 | | presence_penalty | 1.5 | | seed | 3407 |

Reasoner

Reasoner examples produce text outputs from text and vision inputs. It follows Qwen3-VL-compatible message conventions for image and video inputs.

| Workflow | Inputs | Outputs | What it demonstrates | | --- | --- | --- | --- | | Caption | Video | Text | Detailed video captioning | | Temporal localization | Video, query | Text or JSON | Event detection, timestamp query, and interval question answering | | Embodied reasoning | Video, question | Text | Next-action prediction for robotics and assisted-task settings | | Common-sense reasoning | Video, question | Text | Physical common-sense judgment with visible context | | 2D grounding | Image, prompt | JSON boxes | Bounding-box localization from an image prompt | | Describe anything | Image, marked subjects | JSON or text | Attribute captioning for marked subjects | | Action CoT | Image or video, prompt | Text or JSON | Trajectory prediction and driving-scene chain-of-thought | | Physical Plausibility Analysis | Video, prompt | Label | Physical plausibility classification | | Situation Understanding | Video, question | Text | Situation understanding and likely-next-action prediction |

Reasoner examples use the following sampling settings:

| Parameter | Without reasoning | With reasoning | | --- | ---: | ---: | | top_p | 0.8 | 0.95 | | top_k | 20 | 20 | | repetition_penalty | 1.0 | 1.0 | | presence_penalty | 1.5 | 0.0 | | temperature | 0.7 | 0.6 |

Use this basic message shape for text + vision requests:

[
  {
    "role": "system",
    "content": [{"type": "text", "text": "You are a helpful assistant."}]
  },
  {
    "role": "user",
    "content": [
      {"type": "video_url", "video_url": "https://example.com/video.mp4"},
      {"type": "text", "text": "List the notable events with approximate timestamps."}
    ]
  }
]

For explicit reasoning, append this format instruction to the user prompt:

Answer the question using the following format:
Your reasoning.
Write your final answer immediately after the  tag.

Quickstart

Before running examples, create a Hugging Face access token and then authenticate locally:

uvx hf@latest auth login

Set HF_HOME if you want to use a shared cache or a disk with more space. NIM examples use an NGC API key (NGC_API_KEY) instead of Hugging Face authentication.

Generator requires the Guardrail. Request access to the gated nvidia/Cosmos-1.0-Guardrail HF repository for Hugging Face based Generator paths. To disable the guardrail, set enable_safety_checker=False (Diffusers), TRTLLM_DISABLE_COSMOS3_GUARDRAILS=1 or use_guardrails: false through extra_params (TensorRT-LLM), guardrails: false (vLLM-Omni extra_params/extra_args), or --no-guardrails (Cosmos Framework).

Generator with Diffusers

uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate
uv pip install --torch-backend=auto \
  "diffusers @ git+https://github.com/huggingface/diffusers.git" \
  accelerate \
  av \
  cosmos_guardrail \
  huggingface_hub \
  imageio \
  imageio-ffmpeg \
  torch \
  torchvision \
  transformers

import torch
from diffusers import Cosmos3OmniPipeline
from diffusers.schedulers.scheduling_unipc_multistep import UniPCMultistepScheduler
from diffusers.utils import export_to_video
pipe = Cosmos3OmniPipeline.from_pretrained(
    "nvidia/Cosmos3-Nano",
    torch_dtype=torch.bfloat16,
    device_map="cuda",
)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=10.0)
result = pipe(
    prompt="A mobile robot navigates a warehouse aisle and stops at a shelf.",
    negative_prompt="",
    image=None,
    num_frames=189,
    height=720,
    width=1280,
    fps=24,
    num_inference_steps=35,
    guidance_scale=6.0,
    enable_sound=False,
    add_resolution_template=False,
    add_duration_template=False,
    generator=torch.Generator(device="cuda").manual_seed(1234),
)
export_to_video(result.video, "cosmos3_t2v.mp4", fps=24, macro_block_size=1)

Generator with vLLM-Omni

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v "$(pwd):/workspace" \
  -p 8000:8000 \
  --ipc=host \
  -w /workspace \
  vllm/vllm-omni:cosmos3 \
  vllm serve nvidia/Cosmos3-Nano \
  --omni \
  --model-class-name Cosmos3OmniDiffusersPipeline \
  --allowed-local-media-path / \
  --port 8000 \
  --init-timeout 1800

uv venv --python 3.13 --seed --managed-python
source .venv/bin/activate
CUDA 13 driver:
uv pip install --torch-backend=cu130 \
  "vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@main"
CUDA 12.8 driver:
uv pip install --torch-backend=cu128 \
"vllm-omni @ git+https://github.com/vllm-project/vllm-omni.git@main"