Anthropic CEO Dario Amodei has sounded the alarm on the need for urgent research into the interpretability of AI models, highlighting the risks of deploying powerful systems without understanding their inner workings. In a recent essay, Amodei set an ambitious goal for Anthropic to reliably detect most model problems by 2027, acknowledging the significant challenge ahead.

Amodei's concerns are rooted in the rapid advancements being made in AI technology, with models increasingly capable of performing complex tasks but with little understanding of how they arrive at their decisions. This "black box" problem is particularly concerning, as these systems will soon be central to the economy, technology, and national security, and will be capable of significant autonomy.

The issue is exemplified by OpenAI's recent launch of new reasoning AI models, o3 and o4-mini, which perform better on some tasks but also hallucinate more than other models. The company has admitted that it doesn't know why this is happening, highlighting the need for greater understanding of AI decision-making processes.

Anthropic, a pioneer in mechanistic interpretability, has made early breakthroughs in tracing how models arrive at their answers. However, Amodei emphasizes that far more research is needed to decode these systems as they grow more powerful. The company's co-founder, Chris Olah, notes that AI models are "grown more than they are built," with researchers finding ways to improve intelligence but without fully understanding why.

Amodei warns that reaching Artificial General Intelligence (AGI) without understanding how these models work could be dangerous, likening it to creating "a country of geniuses in a data center." He believes that Anthropic's goal of conducting "brain scans" or "MRIs" of state-of-the-art AI models is necessary to identify issues such as lying, seeking power, or other weaknesses. This could take five to ten years to achieve.

Anthropic has made significant investments in interpretability research, including a recent breakthrough in tracing an AI model's thinking pathways through "circuits." The company has identified one circuit that helps AI models understand which U.S. cities are located in which U.S. states, but estimates that there are millions more within AI models.

Amodei is calling on other industry leaders, including OpenAI and Google DeepMind, to increase their research efforts in interpretability. He also suggests that governments should impose "light-touch" regulations to encourage research, such as requirements for companies to disclose their safety and security practices.

Anthropic's focus on safety sets it apart from other tech companies, which have pushed back on California's AI safety bill, SB 1047. The company has issued modest support and recommendations for the bill, which would set safety reporting standards for frontier AI model developers.

In conclusion, Amodei's warning serves as a wake-up call for the tech industry to prioritize understanding AI models before it's too late. As AI continues to advance at an unprecedented rate, it's crucial that researchers and policymakers work together to ensure that these systems are developed and deployed responsibly.