NPR's popular Sunday Puzzle segment, hosted by Will Shortz, has been entertaining listeners for years with its clever brainteasers. But now, a team of researchers has found a new use for these puzzles: testing the limits of artificial intelligence (AI) models. In a recent study, the team created an AI benchmark using riddles from Sunday Puzzle episodes, revealing surprising insights into the strengths and weaknesses of various AI models.

The researchers, hailing from Wellesley College, Oberlin College, the University of Texas at Austin, Northeastern University, Charles University, and startup Cursor, sought to develop a benchmark that would challenge AI models in a more relatable way. Unlike many existing benchmarks that test AI models on esoteric knowledge, the Sunday Puzzle benchmark focuses on problems that humans can understand with general knowledge. According to Arjun Guha, a computer science faculty member at Northeastern and one of the co-authors on the study, this approach allows models to demonstrate their problem-solving abilities without relying on "rote memory."

The Sunday Puzzle benchmark consists of around 600 riddles, which the researchers used to test various AI models. The results showed that reasoning models, such as OpenAI's o1 and DeepSeek's R1, far outperform other models. These models take a more methodical approach, thoroughly fact-checking themselves before providing answers. While this approach takes longer, it helps the models avoid common pitfalls that can trip up AI models.

However, the study also revealed some surprising limitations of these AI models. For instance, DeepSeek's R1 model was found to sometimes "give up" and provide incorrect answers, even stating verbatim "I give up" followed by a random answer. Other models exhibited bizarre behaviors, such as providing wrong answers, retracting them, and then failing again. The researchers also observed models getting stuck in infinite loops, providing nonsensical explanations for their answers, or arriving at correct answers only to consider alternative solutions for no apparent reason.

Guha noted that these behaviors are reminiscent of human frustration, with R1 even literally saying it's getting "frustrated" on hard problems. While it's unclear how this "frustration" affects the quality of model results, it highlights the need for further research into the limitations of AI models.

The current best-performing model on the benchmark is o1, with a score of 59%, followed by o3-mini with a score of 47%. The researchers plan to expand their testing to additional reasoning models, hoping to identify areas where these models can be improved.

The study's findings have significant implications for the AI industry, which is currently facing a benchmarking conundrum. Many existing benchmarks are becoming outdated, and new ones are needed to evaluate AI models in a more realistic and relatable way. By using a public radio quiz game like the Sunday Puzzle, the researchers have demonstrated a promising approach to testing AI models' problem-solving abilities.

As AI models become increasingly deployed in real-world settings, it's essential to develop benchmarks that are accessible and understandable to a broader range of researchers and users. By doing so, we can ensure that these models are capable of solving real-world problems and making a positive impact on society.