high performance computing on graphics processing units: hgpu.org

Graphics Processing Units (GPUs) were originally developed for computer gaming and other graphical tasks, but for many years have been exploited for general purpose computing across a number of areas. They offer advantages over traditional CPUs because they have greater computational capability, and use high-bandwidth memory systems (where memory bandwidth is the main bottleneck for many scientific applications). This Best Practice Guide describes GPUs: it includes information on how to get started with programming GPUs, which cannot be used in isolation but as "accelerators" in conjunction with CPUs, and how to get good performance. Focus is given to NVIDIA GPUs, which are most widespread today. In Section 2, "The GPU Architecture", the GPU architecture is described, with a focus on the latest "Pascal" generation of NVIDIA GPUs, and attention is given to the architectural reasons why GPUs offer performance benefits. This section also includes details of GPU-accelerated services within the PRACE HPC ecosystem. In Section 3, "GPU Programming with CUDA", the NVIDIA CUDA programming model, which includes the necessary extensions to manage parallel execution and data movement, is described, and it is shown how to write a simple CUDA code. Often it is relatively simple to write a working CUDA application, but more work is needed to get good performance. A range of optimisation techniques are presented in Section 4, "Best Practice for Optimizing Codes on GPUs". Large-scale applications will require use of multiple GPUs in parallel: this is addressed in Section 5, "Multi-GPU Programming". Many GPU-enabled libraries exist for common operations: these can facilitate programming in many cases. Some of the popular libraries are described in Section 6, "GPU Libraries". Finally, CUDA is not the only option for programming GPUs and alternative models are described in Section 7, "Other Programming Models for GPUs".