6.4TB SQLite Database Powers Searchcode.com's Upgrade
2025-02-17
Searchcode.com, a source code search engine, migrated its database from MySQL to SQLite, resulting in a massive 6.4TB SQLite database—likely one of the largest publicly facing instances. Driven by a desire for a single binary deployment and reduced dependencies, the migration wasn't without challenges. Issues like database locking and cross-compilation complexities were overcome using dual database connections, a pure Go SQLite version, and filesystem-level compression (BTRFS with zstd). The result? A significantly faster and more scalable Searchcode.com with improved search speed and backend processes.