Noam Brown, the leader of AI reasoning research at OpenAI, has sparked a thought-provoking discussion in the AI community by suggesting that "reasoning" AI models like o1 could have arrived 20 years earlier if researchers had taken the right approach. Speaking at Nvidia's GTC conference in San Jose, Brown emphasized the importance of test-time inference, a technique that enables AI models to "think" before responding to queries, leading to more accurate and reliable results.

Brown, one of the principal architects behind o1, attributed the delay in developing "reasoning" AI models to the neglect of this research direction. He observed that humans often spend a significant amount of time thinking before acting in tough situations, which inspired him to explore the potential of test-time inference in AI. This approach involves applying additional computing to running models to drive a form of "reasoning," making them more accurate and reliable, particularly in domains like mathematics and science.

Despite the significance of test-time inference, Brown acknowledged that pre-training – training ever-larger models on ever-larger datasets – is not obsolete. In fact, AI labs like OpenAI are now splitting their time between pre-training and test-time inference, recognizing the complementary nature of these approaches. This shift in focus highlights the ongoing evolution of AI research, as experts continue to explore new techniques to improve AI models' performance and reliability.

The conversation also touched on the challenges faced by academia in conducting large-scale AI experiments, given the limited access to computing resources. Brown conceded that it has become more difficult for academic institutions to keep pace with AI labs like OpenAI, but he emphasized the opportunities for collaboration and exploration in areas that require less computing, such as model architecture design. This sentiment is particularly relevant in the current climate, where the Trump administration's deep cuts to scientific grant-making have sparked concerns about the impact on AI research efforts.

Brown specifically identified AI benchmarking as an area where academia could make a significant impact, citing the poor state of benchmarks in AI today. He noted that popular AI benchmarks often test for esoteric knowledge and provide scores that correlate poorly to proficiency on tasks that most people care about, leading to widespread confusion about models' capabilities and improvements. By addressing this issue, academia can contribute meaningfully to the development of more effective and reliable AI models.

The comments from Brown, a leading expert in AI reasoning research, offer valuable insights into the current state of AI development and the opportunities for growth. As the AI community continues to navigate the complexities of developing more advanced and reliable models, the importance of collaboration, innovation, and critical evaluation of existing approaches will only continue to grow.