high performance computing on graphics processing units: hgpu.org

We propose a method to parallelize the training of a convolutional neural network by using a CUDA-based cluster. We attain a substantial increase in the performance of the algorithm itself. We research the feasibility of using batch versus online mode training and provide a performance comparison between them. Furthermore, we propose an implementation of an alternative algorithm to compute local gradients which increases the level of parallelism. To conclude, we give a set of best practices for implementing Convolutional Neural Networks on the cluster.