Lessons Learned Optimizing Convolutions with SIMD: Branch Prediction and Compiler Gotchas

The author attempted to optimize convolution operations using SIMD instructions, only to encounter a performance degradation. The initial implementation used SIMD loads, FMA instructions, and loop optimization techniques, but it was more than twice as slow as the unvectorized version. After debugging, the problem was found to be excessive branch instructions causing CPU branch prediction failures, and compiler inlining limitations preventing the proper use of the AVX instruction set. Finally, by reducing branching, splitting loops, and appropriately using compiler inlining attributes, the author successfully improved performance to the expected level. This case study illustrates the complexity of modern CPU architectures and the details that need to be considered in performance optimization.