AI web crawling bots have become a thorn in the side of open source developers, who are disproportionately affected by their aggressive behavior. These bots, often used for search engine optimization, ignore the Robots Exclusion Protocol (robot.txt) file, which tells bots what not to crawl, and can bring down entire websites with their relentless requests. In response, developers are fighting back with cleverness and humor, deploying innovative solutions to block these rogue bots.

One such developer, Xe Iaso, recently shared a "cry for help" blog post detailing the struggles of dealing with AmazonBot, which relentlessly pounded on a Git server website, causing DDoS outages. Iaso lamented that it's futile to block AI crawler bots because they lie, change their user agent, use residential IP addresses as proxies, and more. In response, Iaso created Anubis, a reverse proxy proof-of-work check that must be passed before requests are allowed to hit a Git server. The tool blocks bots but lets through browsers operated by humans, and has become a hit among the FOSS community, collecting 2,000 stars, 20 contributors, and 39 forks on Github.

Anubis is not the only solution being deployed to combat rogue AI crawlers. Other developers are taking a more aggressive approach, such as loading robot.txt forbidden pages with misleading or irrelevant content. This approach, dubbed "vengeance as defense," aims to make it unprofitable for AI crawlers to continue their aggressive behavior. Cloudflare, a commercial player, has also released a tool called AI Labyrinth, which slows down, confuses, and wastes the resources of AI crawlers that don't respect "no crawl" directives.

The issue of rogue AI crawlers has become so pervasive that developers are being forced to take drastic measures to protect their sites. SourceHut's Drew DeVault described spending up to 100% of his time mitigating hyper-aggressive LLM crawlers at scale, while Jonathan Corbet, a famed FOSS developer, warned that his site was being slowed by DDoS-level traffic from AI scraper bots. In some cases, developers have even had to block entire countries from accessing their sites to prevent AI crawlers from causing outages.

The root of the problem lies in the fact that many AI bots don't honor the robot.txt file, which was originally created for search engine bots. This has led to a cat-and-mouse game between developers and AI crawlers, with each side trying to outsmart the other. While solutions like Anubis and AI Labyrinth offer some respite, the issue is unlikely to be resolved until AI crawlers are programmed to respect robot.txt files or alternative solutions are developed.

In the meantime, developers will continue to fight back with creative solutions and a touch of humor. As DeVault pleaded, "Please stop legitimizing LLMs or AI image generators or GitHub Copilot or any of this garbage. I am begging you to stop using them, stop talking about them, stop making new ones, just stop." While this may be a pipe dream, it's clear that the FOSS community will not go down without a fight.

The battle against rogue AI crawlers is a reminder of the importance of responsible AI development and the need for developers to prioritize ethical considerations in their work. As AI technology continues to evolve, it's crucial that we prioritize transparency, accountability, and respect for the digital ecosystem. Only then can we ensure that AI is used for the greater good, rather than causing harm to innocent developers and websites.