9617

Deep learning with COTS HPC systems

Adam Coates, Brody Huval, Tao Wang, David J. Wu, Andrew Y. Ng, Bryan Catanzaro
Stanford University Computer Science Dept., 353 Serra Mall, Stanford, CA 94305 USA
30th International Conference on Machine Learning, 2013

@article{catanzaro2013deep,

   title={Deep learning with COTS HPC systems},

   author={Catanzaro, Bryan},

   year={2013}

}

Download Download (PDF)   View View   Source Source   

1370

views

Scaling up deep learning algorithms has been shown to lead to increased performance in benchmark tasks and to enable discovery of complex high-level features. Recent efforts to train extremely large networks (with over 1 billion parameters) have relied on cloud-like computing infrastructure and thousands of CPU cores. In this paper, we present technical details and results from our own system based on Commodity Off-The-Shelf High Performance Computing (COTS HPC) technology: a cluster of GPU servers with Infiniband interconnects and MPI. Our system is able to train 1 billion parameter networks on just 3 machines in a couple of days, and we show that it can scale to networks with over 11 billion parameters using just 16 machines. As this infrastructure is much more easily marshaled by others, the approach enables much wider-spread research with extremely large neural networks.
No votes yet.
Please wait...

Recent source codes

* * *

* * *

HGPU group © 2010-2017 hgpu.org

All rights belong to the respective authors

Contact us: