NVIDIA has made significant strides in the realm of autonomous vehicle (AV) development with the introduction of the Cosmos world foundation models. These state-of-the-art models, including Predict, Transfer, and Reason, have been specifically crafted to streamline and augment the AV development process. They focus on generating high-quality synthetic data vital for end-to-end AV planning models and tackle the pressing need for rich sensor data.
One of the notable models, Cosmos Transfer, excels in creating diverse driving videos by utilizing HD maps and LIDAR data, alongside a multiview model that ensures consistent representation across multiple camera perspectives. Major players in the AV sector, including companies and simulators like CARLA, Plus, and Oxa, have already started integrating these models to enhance their workflows. The primary objective is to expedite development, adequately address edge cases, and diversify training datasets.
Taking it a notch further, NVIDIA has developed the Cosmos Predict-2 model, a physics-aware synthetic data generator designed for both robotics and AV. Offering two parameter variants, Predict-2 boasts improved speed and visual fidelity. It supports various resolutions and framerates and allows domain-specific customizations through post-training. This flexibility equips developers to create synthetic scenarios tailored to tasks like apple picking in robotics and handling challenging AV scenarios such as rainy highway conditions.
In addition, NVIDIA’s advancements extend to AV simulation. New APIs and tools facilitate acceleration through neural reconstruction and these world foundation models. The NuRec technology converts real-world fleet data into interactive digital twins, complemented by CARLA simulator integration for orchestrating scenarios. Cosmos Transfer plays a pivotal role here, generating diverse synthetic conditions essential for robust simulation.
The development toolkit includes NuRec Fixer for resolving artifacts, a Physical AI Dataset comprising 40,000 clips, and the much-anticipated Omniverse Blueprint for comprehensive simulation pipelines. These tools empower developers to address data coverage gaps and simulate rare events, essential for AV validation and safer deployment.
Extending their collaborative efforts, NVIDIA’s Cosmos models, particularly Predict-2, are being utilized by companies like Plus, Oxa, and Uber to scale AV development. The launch of this model is complemented by the introduction of supporting developer tools, such as the Cosmos Transfer NIM microservice and NuRec Fixer model. As CARLA simulator integrates these innovations for its expansive developer community, NVIDIA’s Halos safety platform is also gaining traction with new partners like Bosch, Easyrain, and Nuro, poised to enhance AV safety verification. This suite of models and tools from NVIDIA is paving the way for accelerated and safer autonomous vehicle development.

Collections

NVIDIA launched Cosmos world foundation models (Predict, Transfer, and Reason) to accelerate autonomous vehicle development through synthetic data generation. The models create diverse driving scenarios using HD maps and LIDAR data, with Cosmos Predict-2 offering physics-aware synthetic data generation for both robotics and AV applications. Major AV companies like Plus, Oxa, and Uber are integrating these tools, while NVIDIA's supporting technologies include NuRec for converting fleet data into digital twins, CARLA simulator integration, and the Halos safety platform for AV verification.

Enhancing Autonomous Vehicle Development with NVIDIA's Cosmos World Foundation Models