Kafka's Unfairness: Why It's a Bad Job Queue at Low Throughput

This article explores the pitfalls of using Kafka as a job queue at low throughput. The author argues that Kafka's underlying mechanism can lead to unfair job distribution, even when other consumers are idle. A single consumer might be overloaded while others sit idle. A formula is provided to calculate the worst-case scenario of jobs assigned to a single consumer. The impact of this unfairness on application performance under varying loads is analyzed. The author concludes that using Kafka as a job queue at low throughput is discouraged unless KIP-932 is implemented.