DualPipe: A Bidirectional Pipeline Parallelism Algorithm for DeepSeek-V3
2025-02-27
The DeepSeek-V3 technical report introduces DualPipe, an innovative bidirectional pipeline parallelism algorithm. DualPipe achieves full overlap of forward and backward computation-communication phases, minimizing pipeline bubbles. This is accomplished through efficient scheduling that interleaves forward and backward computations, significantly improving efficiency. Compared to traditional methods, DualPipe reduces waiting time and memory usage. Developed by Jiashi Li, Chengqi Deng, and Wenfeng Liang.