The Wikimedia Foundation, the organization behind Wikipedia and other crowdsourced knowledge projects, has reported a significant surge in bandwidth consumption on Wikimedia Commons, its repository of open-licensed images, videos, and audio files. According to a blog post, the bandwidth consumption has increased by 50% since January 2024, with automated AI scrapers being the primary cause of this spike.

Contrary to what one might expect, the increased traffic is not due to a growing demand from human users, but rather from data-hungry AI models that rely on scraping data from online sources to train their algorithms. Wikimedia Commons, being a freely accessible repository of multimedia content, has become a prime target for these scrapers, which are designed to extract large amounts of data quickly and efficiently.

The Wikimedia Foundation's infrastructure is built to handle sudden traffic spikes from human users during high-interest events, but the sheer volume of traffic generated by scraper bots is unprecedented and poses significant risks and costs to the organization. According to Wikimedia, almost two-thirds (65%) of the most resource-intensive traffic comes from bots, despite accounting for only 35% of overall pageviews.

The disparity between bot traffic and human traffic can be attributed to the way content is stored and served on Wikimedia Commons. Frequently accessed content is cached closer to the user, reducing the load on the core data center, while less popular content is stored further away, making it more expensive to serve. Bots, which tend to "bulk read" larger numbers of pages, including less popular ones, are more likely to request content from the core data center, thereby increasing the cost of serving that content.

The surge in bot traffic has significant implications for the Wikimedia Foundation, which is now forced to dedicate more resources to blocking crawlers and mitigating the disruption they cause to regular users. Moreover, the increased cloud costs associated with serving this traffic pose a significant financial burden on the organization.

This trend is not unique to Wikimedia Commons, but rather part of a broader issue affecting the open internet. Other developers and publishers are also experiencing similar problems, with some even considering implementing logins and paywalls to protect their resources. The issue has sparked a cat-and-mouse game between developers and AI scrapers, with some tech companies, like Cloudflare, launching innovative solutions, such as AI Labyrinth, to slow down crawlers.

The long-term implications of this trend are concerning, as it could ultimately force many publishers to restrict access to their content, undermining the very principles of the open internet. As the use of AI models continues to grow, it is essential for developers, publishers, and policymakers to work together to find sustainable solutions that balance the needs of AI development with the need to protect the open internet.

In the meantime, the Wikimedia Foundation's experience serves as a stark reminder of the importance of addressing this issue head-on. By highlighting the problem and exploring innovative solutions, we can work towards preserving the open internet and ensuring that it remains a valuable resource for generations to come.