Virtual Threads Ate My Memory: A Web Crawler's Tale

This post details a developer's experience building a web crawler with Java's Virtual Threads. Initially, performance soared, but the crawler crashed with an `OutOfMemoryError`. The author explains how they used Virtual Threads to dramatically increase speed, then shows how they fixed the memory issue by introducing a semaphore to limit concurrent tasks. The comparison between platform threads and Virtual Threads highlights the need for explicit resource management when using Virtual Threads, as the JVM's implicit backpressure mechanisms no longer apply. The author concludes that while Virtual Threads offer significant performance gains, developers must carefully manage resources to avoid memory problems.