In a surprising move, AI research organization Anthropic has used the classic Game Boy game Pokémon Red to benchmark its latest AI model, Claude 3.7 Sonnet. The company's blog post on Monday revealed that the model was equipped with basic memory, screen pixel input, and function calls to press buttons and navigate around the screen, allowing it to play Pokémon continuously.

The unique feature of Claude 3.7 Sonnet is its ability to engage in "extended thinking," similar to other AI models like OpenAI's o3-mini and DeepSeek's R1. This capability enables the model to "reason" through challenging problems by applying more computing power and taking more time. In the context of Pokémon Red, this meant that Claude 3.7 Sonnet could successfully battle three Pokémon gym leaders and win their badges, a significant improvement over its predecessor, Claude 3.0 Sonnet, which failed to leave the starting point in Pallet Town.

While the exact computing resources required to achieve these milestones are unknown, Anthropic disclosed that the model performed 35,000 actions to reach the last gym leader, Surge. This raises questions about the scalability and efficiency of Claude 3.7 Sonnet, which will likely be explored by developers and researchers in the future.

The use of Pokémon Red as a benchmarking tool may seem unconventional, but it is part of a larger trend in the AI research community. Games have long been used to test AI models' abilities, with recent examples including Street Fighter and Pictionary. This approach allows researchers to evaluate their models' decision-making, problem-solving, and adaptability in a controlled environment.

The implications of Claude 3.7 Sonnet's success in Pokémon Red are significant, as it demonstrates the potential for AI models to tackle complex, dynamic problems. As AI continues to advance, we can expect to see more innovative applications of these technologies in various domains. For now, Anthropic's experiment serves as a fascinating example of the creative ways in which AI researchers are pushing the boundaries of what is possible.

Image credits: Anthropic