A recent study has revealed that even the most advanced artificial intelligence (AI) models struggle to answer historical questions accurately, exposing significant limitations in their understanding. The research, presented at the NeurIPS conference, tested three top large language models (LLMs) – OpenAI's GPT-4, Meta's Llama, and Google's Gemini – on historical questions, with disappointing results.

The benchmark, called Hist-LLM, was designed to evaluate the correctness of answers according to the Seshat Global History Databank, a vast database of historical knowledge. The best-performing LLM, GPT-4 Turbo, achieved only about 46% accuracy, barely above random guessing. This raises concerns about the ability of AI models to provide reliable information on complex historical topics.

According to Maria del Rio-Chanona, an associate professor of computer science at University College London and co-author of the paper, "The main takeaway from this study is that LLMs, while impressive, still lack the depth of understanding required for advanced history. They're great for basic facts, but when it comes to more nuanced, PhD-level historical inquiry, they're not yet up to the task."

The researchers shared examples of historical questions that LLMs got wrong, such as whether scale armor was present during a specific time period in ancient Egypt. GPT-4 Turbo incorrectly answered yes, when in fact the technology only appeared in Egypt 1,500 years later. This highlights the models' tendency to extrapolate from prominent historical data, rather than retrieving more obscure knowledge.

Del Rio-Chanona explained that LLMs often rely on publicly available information, which can lead to biases in their responses. For instance, when asked about ancient Egypt's professional standing army, GPT-4 incorrectly answered that it did, likely due to the abundance of information about other ancient empires having standing armies.

The study also identified regional biases in the models' performance, with OpenAI and Llama models performing worse for regions like sub-Saharan Africa. This suggests that the training data used to develop these models may be incomplete or biased, further limiting their ability to provide accurate information.

Despite these limitations, the researchers remain hopeful that LLMs can still aid historians in the future. They are working on refining their benchmark by including more data from underrepresented regions and adding more complex questions. As the paper notes, "Overall, while our results highlight areas where LLMs need improvement, they also underscore the potential for these models to aid in historical research."

The study's findings serve as a reminder that AI models, no matter how advanced, are not yet a substitute for human expertise and critical thinking. While they can excel in certain tasks, such as coding or generating podcasts, they still struggle to demonstrate nuanced understanding in complex domains like history.

As the development of AI continues to accelerate, it is essential to acknowledge and address these limitations to ensure that these models are used responsibly and effectively. By doing so, we can unlock the potential of AI to aid in historical research and other domains, while also promoting a more accurate and informed understanding of the world around us.