AI Crawlers Overwhelm the Open Web: Meta and OpenAI Leading the Charge
2025-08-21
Fastly's report reveals that AI crawlers are consuming the open web at an alarming rate, accounting for 80% of all AI bot traffic. Meta's AI division contributes over half of this crawler traffic, while OpenAI dominates on-demand fetch requests. This excessive scraping leads to increased website load, server overload, and harms content creators. Some companies ignore robots.txt directives, prompting website operators to fight back with anti-scraping techniques like Anubis. Experts call for responsible crawling standards, even suggesting that only the bursting of the AI bubble can solve this, with government regulation becoming urgent.
Tech
web load