Pushing the Limits of Linux Pipes: From 3.5GiB/s to 62.5GiB/s

This post explores the implementation of Unix pipes in Linux by iteratively optimizing a test program that writes and reads data through a pipe. Starting with a simple program achieving around 3.5GiB/s throughput, the author improves its performance twentyfold through several optimization stages. Key improvements include utilizing `vmsplice` and `splice` system calls to eliminate data copying, leveraging huge pages to reduce paging overhead, and employing busy-looping to minimize synchronization costs. The journey is detailed with code examples and performance analysis using Linux's `perf` tool.