Academic Websites Overwhelmed by AI Bot Traffic
2025-06-02
A surge in bot traffic is crippling academic websites. Sites like DiscoverLife, hosting millions of images, have experienced massive traffic spikes, rendering them unusable. The culprit? Bots scraping data, likely to train generative AI models. This isn't isolated; BMJ and Highwire Press report similar issues, with COAR finding over 90% of surveyed members affected, many experiencing service disruptions. While open access encourages reuse, the aggressive scraping is unsustainable. The release of DeepSeek, a less resource-intensive LLM, exacerbated the problem, fueling the bot explosion. Smaller organizations face extinction unless this issue is addressed.