Rust Concurrency Pitfalls: An Atomic Counter Bottleneck

Conviva's streaming analytics platform experienced a performance bottleneck due to a seemingly innocuous atomic counter in a globally shared type registry using a concurrent hash map (Flashmap). Under high concurrency, updates to the atomic counter caused cache line bouncing and excessive context switching, leading to a spike in P99 latency. Replacing Flashmap with Dashmap failed to resolve the issue. The problem was ultimately solved using ArcSwap, which employs a read-copy-update (RCU) mechanism to avoid cache contention. This case highlights the importance of choosing the right data structure for high-concurrency scenarios, particularly in read-heavy situations where ArcSwap's efficiency excels.