NPR's popular Sunday Puzzle segment, hosted by Will Shortz, has been entertaining and challenging listeners for years. But now, a team of researchers has leveraged this unique platform to develop a novel AI benchmark, putting reasoning models to the test and uncovering surprising insights into their problem-solving abilities.

The study, conducted by researchers from Wellesley College, Oberlin College, the University of Texas at Austin, Northeastern University, and startup Cursor, created an AI benchmark using riddles from Sunday Puzzle episodes. According to Arjun Guha, a computer science undergraduate at Northeastern and one of the co-authors on the study, the goal was to develop a benchmark with problems that humans can understand with only general knowledge.

The AI industry is currently facing a benchmarking conundrum, with most tests evaluating skills like PhD-level math and science questions that aren't relevant to the average user. Moreover, many benchmarks are quickly approaching the saturation point. The Sunday Puzzle benchmark offers a refreshing alternative, as it doesn't test for esoteric knowledge and challenges models to think creatively, rather than relying on "rote memory" to solve problems.

Guha explained that what makes these problems hard is that it's difficult to make meaningful progress on a problem until you solve it – that's when everything clicks together all at once. This requires a combination of insight and a process of elimination. While no benchmark is perfect, the Sunday Puzzle benchmark has its limitations, being U.S.-centric and English-only. However, the researchers intend to keep the benchmark fresh and track how model performance changes over time, with new questions released every week.

The study's results showed that reasoning models, such as OpenAI's o1 and DeepSeek's R1, far outperform other models on the benchmark. These models thoroughly fact-check themselves before giving out results, which helps them avoid some of the pitfalls that normally trip up AI models. However, they take a little longer to arrive at solutions – typically seconds to minutes longer.

Interestingly, some models, like DeepSeek's R1, exhibit human-like behavior, such as giving solutions they know to be wrong, stating "I give up," and even expressing "frustration" when faced with challenging problems. This raises questions about how "frustration" in reasoning can affect the quality of model results.

The current best-performing model on the benchmark is o1 with a score of 59%, followed by o3-mini with a score of 47%. As a next step, the researchers plan to broaden their testing to additional reasoning models, which they hope will help identify areas where these models might be enhanced.

Guha emphasized the importance of designing reasoning benchmarks that don't require PhD-level knowledge, making them more accessible to a wider range of researchers. This, in turn, could lead to better solutions in the future. As state-of-the-art models are increasingly deployed in settings that affect everyone, it's crucial that everyone can comprehend and analyze the results, understanding what these models are – and aren't – capable of.

The Sunday Puzzle benchmark has the potential to revolutionize AI benchmarking, providing a more realistic and relatable way to evaluate reasoning models. As the AI industry continues to evolve, this innovative approach could play a significant role in shaping the development of more accurate and effective AI systems.