high performance computing on graphics processing units: hgpu.org

Recent advances in deep learning base on growing model sizes and the necessary scaling of compute power. Training such large-scale models requires an intricate combination of data-, operator-, and pipeline parallelism in complex distributed systems. We show how to use OneFlow’s Split, Broadcast, and Partial Sum (SBP) tensor formulations to enable new distributed training methods with asymptotically optimal communication overheads. Using these insights, we develop AutoDDL, a distributed training framework that combines an exhaustive performance model and automated configuration search to find distributions with near-optimal communication overheads. We conduct evaluations on Multi-Node-Single-GPU and Multi-Node-Multi-GPU machines using different models, including VGG and Transformer. Compared to expert-optimized implementations, AutoDDL reduces the end-to-end training time by up to 31.1% and 10% for Transformer and up to 17.7% and 71.5% for VGG on the two different systems, respectively.