GlassFlow ClickHouse Load Test: Real-Time Deduplication at Scale

GlassFlow conducted a large-scale load test for real-time deduplication, achieving impressive results. On a MacBook Pro, GlassFlow processed over 9,000 records per second from Kafka with sub-0.12ms latency, peaking at 55,000 records per second. Even with 20 million records and 12 concurrent publishers, the system remained robust, with no crashes, message loss, or disordering. The test used synthetic data simulating a real-world use case, evaluating deduplication, throughput, and latency. The full test setup is open-source.