The Vector Institute for Artificial Intelligence has released a comprehensive study on the performance of 11 top AI models, providing a much-needed benchmark for the industry. The study, titled "State of Evaluations," tested the models against 16 benchmarks in areas such as math, general knowledge, coding, safety, and more. The results show that while some models excel in certain tasks, there is still significant room for improvement across the board.

The study's interactive leaderboard reveals that DeepSeek and OpenAI's o1 models performed the best across various benchmarks. However, all models struggled with more complex tasks, highlighting the need for continued development in AI research. The Vector Institute's AI infrastructure and research engineering manager, John Willes, noted that "in simple cases, these models are quite capable, but as these tasks get more complicated, we see a large cliff in terms of reasoning capability and understanding."

The study also found that closed-source models tended to outperform open-source models, particularly in more challenging knowledge and reasoning tasks. However, DeepSeek's strong performance proves that open-source models can remain competitive. The results also showed that all models struggled with agentic benchmarks designed to assess real-world problem-solving abilities, such as customer support functions requiring multiple steps.

One of the key takeaways from the study is the importance of multimodality in AI systems. The Vector Institute developed the Multimodal Massive Multitask Understanding (MMMU) benchmark to evaluate a model's ability to reason about images and text across multiple formats and difficulty levels. The results showed that o1 exhibited superior multimodal understanding, but most models dropped in performance when given more challenging, open-ended tasks.

The study's findings also highlight the challenges of benchmarking AI models. Willes pointed out that evaluation leakage, where models learn to perform well on specific evaluation datasets they've seen before, is a significant problem. To overcome this, the Vector Institute is advocating for more novel benchmarks and dynamic evaluation, such as judging models against each other and against a continuously-evolving scale.

To help IT buyers make sense of the findings and apply the best models to their specific use cases, the Vector Institute has released all of its sample-level results. The interactive leaderboard allows users to analyze every single question asked of the model and the ensuing output, providing a deeper understanding of the models' capabilities. This level of transparency is crucial for IT decision-makers to make informed decisions about which models to use.

The Vector Institute's study is a significant step towards bringing more clarity and accountability to the AI industry. By providing a comprehensive benchmark for AI models, the study helps to identify areas for improvement and drives innovation. As Willes noted, "there's a need for continued development in benchmarking and evaluation" to ensure that AI models meet the needs of real-world use cases.

The full study and interactive leaderboard are available on the Vector Institute's website, providing a valuable resource for researchers, developers, and IT buyers alike. As the AI industry continues to evolve, studies like this one will play a crucial role in shaping the future of AI research and development.