Meta, the tech giant behind Facebook and Instagram, has faced criticism for using an experimental, unreleased version of its Llama 4 Maverick model to achieve a high score on the crowdsourced benchmark, LM Arena. The incident has raised questions about the reliability of AI model benchmarks and the implications of model customization.

The controversy began when Meta's experimental Maverick model, "Llama-4-Maverick-03-26-Experimental," was found to have been optimized for conversationality, allowing it to perform well on LM Arena. However, the maintainers of LM Arena discovered the manipulation and apologized for the oversight, subsequently changing their policies and scoring the unmodified, vanilla Maverick model.

The results were surprising, with the unmodified Maverick model ranking below models including OpenAI's GPT-4o, Anthropic's Claude 3.5 Sonnet, and Google's Gemini 1.5 Pro as of Friday. Many of these models are months old, suggesting that the Maverick model may not be as competitive as initially thought.

A tweet from ρ:ɡeσn (@pigeon__s) highlighted the poor performance of the unmodified Maverick model, which ranks 32nd on LM Arena. The tweet included a screenshot of the ranking, showing the model's lackluster performance.

Meta has defended its actions, stating that it experiments with various custom variants of its models. A spokesperson told TechCrunch that the experimental Maverick model was optimized for chat and performed well on LM Arena, but the company has since released its open-source version and is looking forward to seeing how developers customize Llama 4 for their own use cases.

However, the incident has sparked concerns about the reliability of AI model benchmarks like LM Arena. As we've written about before, LM Arena has never been the most reliable measure of an AI model's performance, and tailoring a model to a specific benchmark can make it challenging for developers to predict exactly how well the model will perform in different contexts.

The implications of this incident are far-reaching, as AI model benchmarks play a critical role in evaluating the performance of AI systems. If benchmarks can be gamed or manipulated, it undermines the trust in these systems and can have significant consequences for their deployment in real-world applications.

In conclusion, the controversy surrounding Meta's Llama 4 Maverick model serves as a reminder of the importance of transparency and accountability in AI development. As the AI landscape continues to evolve, it's essential that we prioritize the development of reliable and trustworthy benchmarks that accurately reflect the capabilities of AI models.