OpenAI CEO Sam Altman recently sparked a debate among experts when he stated that the "IQ" of AI has been rapidly improving over the past several years. Altman likened the progress to moving one standard deviation of IQ every year, although he acknowledged that this was not a scientifically accurate measurement. However, many experts argue that using IQ as a benchmark for AI progress is misleading and inaccurate.

IQ tests, which are designed to measure human intelligence, are relative and not objective measures of certain kinds of intelligence. They are primarily used to evaluate logic and abstract reasoning, but do not account for practical intelligence or other aspects of human cognition. Moreover, IQ tests have been criticized for their potential biases, as they are often based on Western cultural norms and require a strong working memory.

Experts warn that AI models may have an unfair advantage on IQ tests due to their massive amounts of memory and internalized knowledge. Additionally, AI models are often trained on public web data, which includes example questions from IQ tests. This means that AI models may be able to "game" the system and achieve high scores without truly demonstrating intelligence.

Os Keyes, a doctorate candidate at the University of Washington, notes that IQ tests are "pretty easy to game if you have a practically infinite amount of memory and patience." Mike Cook, a research fellow at King's College London, adds that IQ tests are designed for humans and are not suitable for evaluating AI capabilities. "A crow might be able to use a tool to recover a treat from a box, but that doesn't mean it can enroll at Harvard," Cook said.

The limitations of IQ tests highlight the need for better AI tests, according to Heidy Khlaaf, chief AI scientist at the AI Now Institute. Khlaaf argues that the nature of computation means that systems have always been able to complete tasks beyond human ability, and that directly comparing AI performance to human abilities is a recent and contested phenomenon.

As AI continues to advance, it is essential to develop more accurate and nuanced measures of its capabilities. Relying on IQ tests may provide a misleading picture of AI's progress and potential. Instead, experts recommend exploring alternative evaluation methods that can better capture the complexities of AI's abilities and limitations.

In conclusion, while Altman's comments may have sparked excitement about AI's rapid progress, they also underscore the need for a more thoughtful and nuanced approach to evaluating AI capabilities. By recognizing the limitations of IQ tests and developing more suitable evaluation methods, we can better understand the true potential and implications of AI.