The current state of Physical AI development is a logistical fragmentation. Engineering teams typically operate three distinct GPU silos: one for generating synthetic data, one for post-training the policy, and a third for simulation and evaluation. This fragmented lifecycle forces developers to constantly provision and tear down infrastructure, moving massive datasets across different regions or availability zones, which creates immense operational overhead and resource waste. The industry has been waiting for a way to collapse these disparate stages into a single, fluid pipeline where the model itself handles the transition from simulation to execution.
The Cosmos 3 Hierarchy and Omnimodal Architecture
NVIDIA addresses this fragmentation with Cosmos 3, an omnimodal world foundation model designed to treat video, images, actions, and sound as a single, continuous token stream. Rather than deploying a suite of specialized models, Cosmos 3 utilizes a unified transformer trunk that can be toggled between different operational modes. To ensure broad accessibility and industry adoption, NVIDIA has released the model under the Linux Foundation's OpenMDW-1.1 license.
The model family is structured into three distinct tiers based on computational scale and deployment targets. At the top is Cosmos3-Super, a 64B parameter powerhouse built upon a 32B Qwen3-VL backbone. This model serves as the primary engine for high-fidelity data generation and complex reasoning. Below it sits Cosmos3-Nano, a 16B parameter model utilizing an 8B Qwen3-VL backbone, designed for efficient post-training and deployment. This hierarchical structure allows developers to implement a teacher-student framework, where the Super model generates high-quality synthetic data to train the Nano model.
For the edge, NVIDIA introduced Cosmos3-Edge, a 4B parameter model optimized for embedded hardware like Jetson Thor and Orin. Unlike the Nano and Super tiers, the Edge model is not initialized from Qwen3-VL; it features a separate weight lineage based on a 2B backbone, specifically post-trained to meet the strict constraints of on-device inference. Additionally, for specialized robotics tasks, the Cosmos3-Nano-Policy-DROID variant provides a control update frequency of 15Hz and a prediction horizon of 32 steps, defining the range of future actions the model can forecast in a single pass.
The AR-DM Integration and the Three-Mode Shift
The true technical breakthrough of Cosmos 3 lies in its integration of Autoregressive (AR) and Diffusion Model (DM) structures within a single transformer trunk. The model processes text and vision tokens through an AR subsequence for understanding, while utilizing a DM subsequence to generate video, audio, and action tokens. These are unified through shared Reasoner and Generator towers. To maintain logical consistency, the architecture employs an asymmetric attention mechanism: DM queries can reference both AR and DM keys via full attention, but AR queries remain causal, meaning they cannot see the diffusion tokens. This ensures the model fully understands the input context before attempting to generate a physical response.
This unified architecture allows the model to switch between three critical operational modes simply by altering how noise tokens are configured. In Forward-dynamics mode, the model fixes the current frame and the intended action, then denoises the resulting future video. This effectively turns Cosmos 3 into a world simulator that predicts how an environment will react to a specific robot movement. Conversely, Inverse-dynamics mode fixes the video data and denoises the action tokens required to produce that movement. This transforms the model into an automated labeling tool, capable of extracting precise action labels from massive amounts of unlabeled video data.
Finally, Policy mode fixes only the initial frame and denoises the optimal sequence of actions the robot should take to reach a goal. While the first two modes focus on data preparation and simulation, Policy mode is the execution phase. By housing all three modes in one model, NVIDIA ensures that the physical understanding gained during the simulation and labeling phases directly enhances the performance of the final policy model. The result is a streamlined pipeline where the generative capabilities of the world model act as a direct catalyst for robotic precision.
Optimizing GPU Goodput via SageMaker HyperPod
Moving from a static training job to a continuous Physical AI pipeline shifts the primary success metric from raw throughput to GPU Goodput. GPU Goodput measures the actual progress made across the entire closed-loop pipeline per reserved GPU hour. In traditional setups, the time spent moving data between clusters or re-provisioning nodes creates significant idle time, dragging down the overall Goodput. To solve this, Cosmos 3 is designed to operate within a single, time-shared GPU pool where generation, training, and evaluation happen on the same infrastructure.
Because the 64B Mixture of Transformers (MoT) architecture requires massive parameter synchronization and handles long sequence lengths, it demands extremely low-latency interconnects to avoid communication bottlenecks. Amazon SageMaker HyperPod on Amazon EKS provides this environment, offering a single control plane to manage multi-node jobs while optimizing the physical network layer. This setup eliminates the need to move weights and data across different clusters, allowing the model factory to operate as a continuous loop.
This operational flow is managed through the `awsome-distributed-ai` GitHub repository (https://github.com/aws-samples/awsome-distributed-ai), which provides the infrastructure templates and manifests needed to automate the closed-loop system. The process begins by storing raw data—such as DROID, BridgeData2, or AV sensor logs—in Amazon S3 and Amazon FSx for Lustre to ensure high-speed data throughput. The Cosmos3-Super model then accesses this corpus to generate synthetic data, filling gaps in the dataset with rare or difficult edge cases.
Once the corpus is augmented, the Cosmos3-Nano policy model undergoes post-training, with both the Nano and Super tiers receiving vision fine-tuning to sharpen perception. The final stage is a closed-loop simulation where the policy is tested. Any failures encountered during simulation are fed back into the second stage as targets for the Super model, which then generates specific synthetic data to correct those failures. This creates a self-improving flywheel where the infrastructure's speed and the model's versatility combine to accelerate the path to deployment.
By collapsing the boundaries between the simulator, the labeler, and the actor, NVIDIA has turned the AI training process into a factory line. The focus is no longer on the efficiency of a single training run, but on the velocity of the entire feedback loop.
