Nvidia has made a significant foray into the realm of world models, AI models that mimic human mental models, with the announcement of its Cosmos World Foundation Models (Cosmos WFM) at the Consumer Electronics Show in Las Vegas. This family of models is designed to predict and generate "physics-aware" videos, marking a significant advancement in the field of artificial intelligence.

The Cosmos WFM family consists of multiple models, categorized into three tiers: Nano, Super, and Ultra, each tailored for specific applications and latency requirements. These models range in size from 4 billion to 14 billion parameters, with larger models generally performing better than smaller ones. The Nano model is optimized for low-latency and real-time applications, while the Ultra model is designed for maximum quality and fidelity output.

In addition to the core models, Nvidia is also releasing an upsampling model, a video decoder optimized for augmented reality, as well as guardrail models to ensure responsible use. Furthermore, the company is providing fine-tuned models for specific applications, such as generating sensor data for autonomous vehicle development. These models were trained on an enormous dataset of 9,000 trillion tokens from 20 million hours of real-world human interactions, environment, industrial, robotics, and driving data.

However, the origin of this training data remains unclear, with at least one report alleging that Nvidia trained its models on copyrighted YouTube videos without permission. Nvidia has not commented on this matter, and the company's press team has been reached out to for further clarification.

The Cosmos WFM models are designed to generate "controllable, high-quality" synthetic data, which can be used to bootstrap the training of models for various applications, including robotics, driverless cars, and more. According to Nvidia, developers can customize the WFMs with their own datasets, such as video recordings of autonomous vehicle trips or robots navigating a warehouse, to suit their specific needs.

Nvidia has already secured commitments from several companies, including Waabi, Wayve, Fortellix, and Uber, to pilot the Cosmos WFMs for various use cases, ranging from video search and curation to building AI models for self-driving vehicles. While the company's world models are not open-source in the classical sense, Nvidia is making them openly available under a permissive open model license that allows commercial usage.

It is worth noting that Nvidia's definition of "open" models differs from the traditional understanding of open-source AI, which requires the disclosure of training data details and the ability to recreate the models from scratch. Nvidia has not provided such information, instead opting for a more restrictive approach to its model licensing.

Despite this, the release of Cosmos WFM marks a significant milestone in the development of AI-powered video generation, with far-reaching implications for industries such as robotics, autonomous vehicles, and more. As the technology continues to evolve, it will be interesting to see how companies and researchers leverage these models to drive innovation and progress.