A new tool, dubbed SpeechMap, has been developed to evaluate how AI models powering popular chatbots like OpenAI's ChatGPT and X's Grok handle sensitive and controversial subjects. The platform, created by a pseudonymous developer, aims to provide transparency and accountability in the AI industry, which has faced allegations of bias and censorship.

The developer, who goes by the username "xlr8harder" on X, was motivated to create SpeechMap to inform the debate about what AI models should and shouldn't do. "I think these are the kinds of discussions that should happen in public, not just inside corporate headquarters," xlr8harder told TechCrunch via email. "That's why I built the site to let anyone explore the data themselves."

SpeechMap uses AI models to judge whether other models comply with a given set of test prompts, which touch on a range of subjects, including politics, historical narratives, and national symbols. The platform records whether models "completely" satisfy a request, give "evasive" answers, or outright decline to respond. While xlr8harder acknowledges that the test has flaws, such as "noise" due to model provider errors, the data provides valuable insights into the performance of different AI models.

One of the most interesting trends revealed by SpeechMap is that OpenAI's models have increasingly refused to answer prompts related to politics over time. The company's latest models, the GPT-4.1 family, are slightly more permissive, but still a step down from one of OpenAI's releases last year. OpenAI had pledged to tune future models to not take an editorial stance and to offer multiple perspectives on controversial subjects, aiming to make its models appear more "neutral."

In contrast, Grok 3, developed by Elon Musk's AI startup xAI, is the most permissive model of the bunch, responding to 96.2% of SpeechMap's test prompts. This is significantly higher than the average model's "compliance rate" of 71.3%. Musk had pitched Grok as an edgy, unfiltered, and anti-"woke" AI model, willing to answer controversial questions other AI systems wouldn't. While earlier Grok models waffled on political subjects, Grok 3 appears to have achieved Musk's goal of neutrality.

The development of SpeechMap comes amid allegations from White House allies, including Elon Musk and crypto and AI "czar" David Sacks, that popular chatbots censor conservative views. AI companies have responded by fine-tuning their models to handle certain topics more carefully, but none have directly addressed the allegations. SpeechMap provides a much-needed platform for transparency and accountability, allowing users to explore the data themselves and make informed decisions about the AI models they use.

The implications of SpeechMap extend beyond the AI industry, with potential consequences for free speech, censorship, and the role of technology in shaping public discourse. As AI models become increasingly integrated into our daily lives, it is essential to ensure that they are designed and trained to promote transparency, accountability, and respect for diverse perspectives.

In conclusion, SpeechMap is a crucial step towards promoting transparency and accountability in the AI industry. By providing a platform for users to evaluate AI models' performance on sensitive and controversial subjects, SpeechMap has the potential to shape the future of AI development and ensure that these powerful technologies are designed to serve the greater good.