high performance computing on graphics processing units: hgpu.org

We present a system that enables simple and intuitive programming of CPU+GPU clusters. This system relieves the programmer of the burden of load balancing, detailed data communication, task mapping, scheduling, etc. Our programming model is based on bulk synchronous distributed shared memory model, which is suitable for heterogenous multi-GPU clusters, especially so for compute intensive workloads. We report prototype applications using our system. For example, sequential version of matrix multiplication or 2D FFT requires about 30 additional lines of code to parallelize on a cluster. Distributing multiplication of two square matrices, with 1 billion elements each, across a small cluster with 120 CPU cores and 20 GPUs, our runtime scheduler achieves more than 140x speedup over the single core CPU implementation; the single GPU implementation runs out of memory for this experiment. This performance is possible due to a number of challenging optimizations working in concert. These include prefetching, pipelining, maximizing overlap between computation and communication, and scheduling across devices of vastly different capacities.