The artificial intelligence community on X has been abuzz with a new, informal benchmark that tests AI models' ability to handle complex prompts. The "ball in rotating shape" benchmark, which involves writing a Python script for a bouncing yellow ball within a shape that slowly rotates, has sparked a frenzy of interest and debate among AI enthusiasts.

The benchmark has been used to compare the performance of various AI models, including those from Chinese AI lab DeepSeek, OpenAI, Anthropic, and Google. According to users on X, DeepSeek's freely available R1 model has outperformed OpenAI's o1 pro mode, which costs $200 per month as part of OpenAI's ChatGPT Pro plan. Meanwhile, Anthropic's Claude 3.5 Sonnet and Google's Gemini 1.5 Pro models have struggled with the task, resulting in the ball escaping the shape.

However, experts are questioning the relevance and usefulness of this benchmark in measuring AI models' capabilities. Simulating a bouncing ball is a classic programming challenge that requires accurate collision detection algorithms, but it may not be a very empirical AI benchmark. Even slight variations in the prompt can yield different outcomes, making it difficult to draw meaningful conclusions.

One researcher, n8programs, a researcher in residence at AI startup Nous Research, has pointed out that it took him roughly two hours to program a bouncing ball in a rotating heptagon from scratch. "One has to track multiple coordinate systems, how the collisions are done in each system, and design the code from the beginning to be robust," n8programs explained. While this benchmark may be a reasonable test of programming skills, it may not be a reliable indicator of AI models' capabilities.

The "ball in rotating shape" benchmark highlights the intractable problem of creating useful systems of measurement for AI models. It's often difficult to tell what differentiates one model from another, outside of esoteric benchmarks that don't resonate with most people. Many efforts are underway to build better tests, such as the ARC-AGI benchmark and Humanity's Last Exam, which may provide more meaningful insights into AI models' capabilities.

In the meantime, the AI community will continue to watch with fascination as balls bounce in rotating shapes, but it's clear that more work needs to be done to develop benchmarks that truly measure AI models' abilities.