Distributed Training Large-Scale Deep Architectures

Shang-Xuan Zou, Chun-Yen Chen, Jui-Lin Wu, Chun-Nan Chou, Chia-Chin Tsao, Kuan-Chieh Tung, Ting-Wei Lin, Cheng-Lung Sung, Edward Y. Chang
HTC AI Research, Taipei, Taiwan
arXiv:1709.06622 [cs.DC], (10 Aug 2017)


   title={Distributed Training Large-Scale Deep Architectures},

   author={Zou, Shang-Xuan and Chen, Chun-Yen and Wu, Jui-Lin and Chou, Chun-Nan and Tsao, Chia-Chin and Tung, Kuan-Chieh and Lin, Ting-Wei and Sung, Cheng-Lung and Chang, Edward Y.},






Download Download (PDF)   View View   Source Source   



Scale of data and scale of computation infrastructures together enable the current deep learning renaissance. However, training large-scale deep architectures demands both algorithmic improvement and careful system configuration. In this paper, we focus on employing the system approach to speed up large-scale training. Via lessons learned from our routine benchmarking effort, we first identify bottlenecks and overheads that hinter data parallelism. We then devise guidelines that help practitioners to configure an effective system and fine-tune parameters to achieve desired speedup. Specifically, we develop a procedure for setting minibatch size and choosing computation algorithms. We also derive lemmas for determining the quantity of key components such as the number of GPUs and parameter servers. Experiments and examples show that these guidelines help effectively speed up large-scale deep learning training.
Rating: 3.5/5. From 2 votes.
Please wait...

* * *

* * *

HGPU group © 2010-2019 hgpu.org

All rights belong to the respective authors

Contact us: