high performance computing on graphics processing units: hgpu.org

The development of parallel architectures is now nearly ubiquitous in not only the high-performance computing field, but also the commodity electronics market. Even embedded processors found in cell phones and tablet computers are starting to incorporate parallel architectures. These architectures are exploiting both SIMD (Single-Instruction Multiple-Data) and SIMT (Simple- Instruction Multiple-Thread) parallelism to achieve higher levels of performance that was previously possible. Additionally, multiprocessors are becoming increasingly heterogeneous by incorporating different architectures into the same die, such as NVIDIA’s Tegra and AMD’s Fusion APU. As the computer hardware industry moves to increasingly parallel and heterogeneous architectures, the computer software industry has been forced to make drastic changes to the way software is designed and developed. This has led to an increasing burden on software developers to not only write bug-free software, but also to scale software performance with the diverse architectures. Multi-processors are increasingly exposing SIMD parallelism as a way to improve the per-core performance of the hardware without requiring significant clock speed increases. Vector instruction sets are using larger vector widths to increase the parallelism available. Intel’s SSE instruction set on x86 uses 128-bit vector and instructions that can operate on 4 single-precision or 2 double-precision floating-point numbers at a time. Recently, the AVX instruction set on x86 extends to 256-bit vectors and instructions that can operate on 8 single-precision or 4 double-precision floating-point numbers at a time. Exploiting the SIMD parallelism available in modern hardware can be a difficult process for software developers. Vector instruction sets often impose limitations such as alignment restrictions on vector loads/stores and a lack of scatter/gather operations that make it a non-trivial process to convert scalar code into higher-performance vector code. In the first part of this dissertation, we present a method for automatically finding sections of scalar application code that would likely benefit from SIMD vectorization. Many-core architectures such as those found in Graphics Processing Units (GPUs) have become good targets for high-performance applications such as those found in scientific computing. Modern high-end GPUs have a theoretical floating-point throughput of over 2 TFlop/s, making them prime targets for scientific computing. Programming environments such as NVIDIA’s CUDA, Khronos’ OpenCL, and Microsoft’s DirectCompute allow application developers to write software that executes directly on GPU hardware, but their abstractions are very close to the actual hardware and are complex for developers to use. To take advantage of GPU devices, application developers must first determine which parts of their applications would benefit from GPU acceleration, then port those parts of their application to CUDA, OpenCL, or DirectCompute. After porting their code, significant optimization effort is then often required to maximize performance of the code, separate from any previous effort spent on optimization of the GPU versions. This makes GPU programming very complex for computational scientists and other software writers that do not have a background in computer architecture and GPU programming models. In the second part of this dissertation, we present an automatic code generation framework for stencil computations on GPU devices. Stencil computations are important parts of many scientific applications, including PDE solvers, grid-based simulations, and image processing. Our code generation framework takes a high- level description of the stencil problem and generates high-performance code for a variety of GPU architectures. The performance of GPU programs is often highly dependent on the choice of thread block and tile size. The optimal choice of block and tile size can be different based on the characteristics of the GPU program. The code generation framework for stencil programs proposed in this work is parameterized on the choice of block and tile size, and the choice of block and tile size can have a large impact on performance. In the third part of this dissertation, we explore the effects of the choice of block and tile size on the performance of stencil programs generated by our code generation framework and propose a performance model that uses a description of the stencil program and the target GPU hardware to automatically select an optimal block and tile size.