Distributed Training of Deep Neuronal Networks: Theoretical and Practical Limits of Parallel Scalability
Fraunhofer ITWM
arXiv:1609.06870 [cs.CV], (22 Sep 2016)
@article{keuper2016distributed,
title={Distributed Training of Deep Neuronal Networks: Theoretical and Practical Limits of Parallel Scalability},
author={Keuper, Janis},
year={2016},
month={sep},
archivePrefix={"arXiv"},
primaryClass={cs.CV}
}
This paper presents a theoretical analysis and practical evaluation of the main bottlenecks towards a scalable distributed solution for the training of Deep Neuronal Networks (DNNs). The presented results show, that the current state of the art approach, using data-parallelized Stochastic Gradient Descent (SGD), is quickly turning into a vastly communication bound problem. In addition, we present simple but fixed theoretic constraints, preventing effective scaling of DNN training beyond only a few dozen nodes. This leads to poor scalability of DNN training in most practical scenarios.
September 30, 2016 by hgpu
