Rust: Investigating a Mysterious OOM
2025-01-19

Qovery's engine-gateway, a Rust service, experienced unexpected out-of-memory (OOM) crashes. Monitoring showed stable memory usage before abrupt restarts. The culprit? The anyhow library, when backtraces are enabled, captures a backtrace for every error. Symbol resolution, only triggered when printing errors in debug mode (`{:?}`), caused massive memory consumption. Setting environment variables `RUST_BACKTRACE=1` and `RUST_LIB_BACKTRACE=0` to enable backtraces only on panic solved the issue. This highlights how monitoring can be deceptive and the importance of thorough library documentation review.
Development
OOM