Eliminating Noise in CI Performance Testing: The CodSpeed Macro Runners Breakthrough

Creating performance gates in CI to prevent significant regressions has been a challenge due to noise in hosted runners. This article explores measuring this noise using various benchmarking suites. Results on GitHub Actions showed a 2.66% coefficient of variation, leading to a 45% false positive rate for a 2% performance gate. CodSpeed's Macro Runners, running on bare-metal cloud instances with enhanced stability, drastically reduced this noise. Macro Runners achieved a 0.56% average variance, lowering the false positive rate to 0.04%. This allows for more precise performance gates, catching subtle regressions without overwhelming contributors with false alarms.