Fighting Back Against Abusive Web Crawlers with Fake JPEGs
2025-07-12
The author developed a small web application called Spigot that generates fake web pages to confuse malicious web crawlers. Recently, a crawler called ImageSiftBot aggressively targeted it for images, despite Spigot lacking them. To counter this, the author cleverly exploits the structure of JPEG files, generating fake JPEGs filled with random data. This effectively wastes the crawler's resources while minimizing CPU load on the server. The approach is simple and efficient, requiring minimal code, and the generated images display correctly in most browsers.
Read more
Development
malicious crawlers