Web Server Listen Overflows Traced to a Linux Kernel Performance Issue

2025-02-14

Upgrading web servers from CentOS to Ubuntu led to listen overflow errors. Investigation revealed a system CPU spike on newly booted Ubuntu hosts within minutes of startup, causing slow web request processing and subsequent listen overflows. The culprit was inode cgroup switching in the Linux kernel; after writing many files, the kernel spent significant time moving inodes between cgroups. Disabling the io or memory controllers in systemd resolved the issue. CentOS was unaffected as it uses cgroups v1, unlike Ubuntu's cgroups v2. A minimal reproduction script was created to demonstrate the issue.

Development Performance Issue