Microsoft has made a significant breakthrough in AI for gaming, unveiling a new model that can generate gameplay and potentially revolutionize the industry. The company's research arm, Microsoft Research, has partnered with Xbox game studio Ninja Theory to create the Muse AI model, which can generate a game environment based on visuals or players' controller actions.

The Muse model is a first-of-its-kind generative AI model that understands 3D game worlds and game physics, allowing it to react to how players interact with a game. This enables the model to create consistent and diverse gameplay rendered by AI, demonstrating a major step toward generative AI models that can empower game creators. According to Fatima Kardar, corporate vice president of gaming AI at Microsoft, "This allows the model to create consistent and diverse gameplay rendered by AI, demonstrating a major step toward generative AI models that can empower game creators."

The Muse model was trained on a large amount of human gameplay data from the Xbox game Bleeding Edge, thanks to the collaboration between Microsoft Research and Ninja Theory. This partnership allowed the model to access the equivalent of seven years of human gameplay, resulting in the model being trained on a billion image action pairs in total. Although the model is still in its early stages, with a current resolution of 300x180 pixels, it has the potential to greatly enhance game development and preservation.

Microsoft has released examples of the Muse model in action, generating gameplay and even allowing players to load visual elements into the game to prompt the model. While the company stresses that this technology is not designed to generate an entire game and replace creators, it could be used to help with the early phases of game development. Developers might be able to use Muse in the future to help with prototyping games or adding new AI-powered experiences to existing titles.

During a press briefing, Katja Hofmann, head of Microsoft Research's game intelligence team, demonstrated a real-time version of Muse that Microsoft is currently experimenting with to enable interactive AI-powered games. The demo generated game visuals on the fly in real time and even reacted to objects being dropped into the game to change the environment. Although the real-time gameplay only ran at 10fps and a 300x180 resolution, it was an early demonstration of what might be possible in the future.

Microsoft is also exploring how Muse could help improve classic games and bring them to modern hardware. According to Microsoft Gaming CEO Phil Spencer, "You could imagine a world where from gameplay data and video that a model could learn old games and really make them portable to any platform where these models could run." This could potentially revolutionize game preservation, allowing classic games to be played on modern consoles and PCs.

Despite the potential benefits of AI in gaming, there are concerns among game developers and studios about how AI could affect the creative process of making games. However, Ninja Theory studio head Dom Matthews emphasized that "we don't intend to use this technology for the creation of content. I think the interesting aspect for us that's exciting, is how can we use technology like this to make the process of making games quicker and easier for our talented team, so that they can really focus on the thing that's really special about games: the human creativity."

Microsoft is planning to share AI tools and experiments with Xbox players and creators "earlier on" so that it can "make sure that our AI innovations address real problems and add new value to creating or playing with Xbox." The company is also making it clear that it's up to individual studios at Xbox how they want to use AI in games, with Fatima Kardar stating that "as part of this, we have empowered creative leaders here at Xbox to decide on the use of generative AI. There isn't going to be a single solution for every game or project, and the approach will be based on the creative vision and goals of each team."

As Microsoft continues to explore the potential of AI in gaming, it will be interesting to see how this technology develops and is adopted by the gaming industry. With its potential to enhance game development, preservation, and creativity, the Muse AI model could be a game-changer for the industry.