Website operators are taking drastic measures against aggressive AI web crawlers that are overwhelming their infrastructure, including blocking entire countries and implementing computational puzzles for access.
Software developer Xe Iaso recently faced repeated service outages when Amazon's AI crawlers flooded their Git repository, evading standard blocking methods by spoofing user agents and cycling through residential IP addresses. This led Iaso to create "Anubis," a system requiring visitors to solve computational puzzles before accessing content.
The issue extends beyond individual cases. According to LibreNews, some open source projects report that AI company bots generate up to 97% of their traffic. This has dramatically increased costs and created stability issues for community-maintained infrastructure.
The Fedora Pagure project blocked all traffic from Brazil after failing to control bot activity. When GNOME GitLab implemented Anubis, only 3.2% of requests passed the challenge system, revealing the massive scale of automated traffic. KDE's GitLab temporarily went offline due to crawler traffic from Alibaba IP ranges.
While effective at filtering bots, these defensive measures impact legitimate users. Mobile visitors report waiting up to two minutes to complete proof-of-work challenges. When many users access the same link simultaneously, significant delays occur.
The financial impact is substantial. Read the Docs saved approximately $1,500 monthly in bandwidth costs after blocking AI crawlers, reducing daily traffic from 800GB to 200GB.
Open source projects face particular challenges as they operate with limited resources. Maintainers report that AI crawlers deliberately bypass standard blocking measures by ignoring robots.txt directives and rotating IP addresses. Some projects now receive AI-generated bug reports containing fabricated vulnerabilities, wasting developer time.
Traffic analysis shows varying levels of crawler activity from different companies. Diaspora found that OpenAI accounted for about 25% of web traffic, Amazon 15%, and Anthropic 4.3%. The crawlers revisit pages every few hours, suggesting ongoing data collection rather than one-time training.
New defensive tools are emerging in response. An anonymous creator developed "Nepenthes" to trap crawlers in endless mazes of fake content, while Cloudflare recently launched "AI Labyrinth" as a commercial solution. The ai.robots.txt project maintains an open list of AI crawlers and provides blocking configurations.
Without industry self-regulation or meaningful oversight, the conflict between AI companies and infrastructure maintainers appears set to escalate, potentially threatening the stability of critical online resources.