Building Databases on Object Storage: Taming High Latency

This post delves into common challenges, particularly high latency, encountered when building databases on object storage like AWS S3. The author highlights that object storage latency often follows a lognormal distribution, with long tail latencies significantly impacting performance. To mitigate this, three strategies are proposed: request hedging (sending multiple requests and taking the fastest response), latency-based retrying (retrying after exceeding a threshold), and using different endpoints. Furthermore, the post advocates for caching and horizontal scaling to further optimize performance. Caching reduces the number of object storage accesses, while horizontal scaling leverages the range read API of object storage to boost throughput. The author emphasizes that the optimal strategy depends on the specific application and cost considerations. A Rust program demonstrating these strategies is also mentioned.