high performance computing on graphics processing units: hgpu.org

Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, Matei Zaharia

View

Download (PDF)

Source codes

Tags: Computer science, CUDA, Data parallelism, Deep learning, GPU cluster, NLP, nVidia, nVidia DGX-A100, Package

April 18, 2021 by hgpu

A Hybrid Parallelization Approach for Distributed and Scalable Deep Learning

Samson B. Akintoye, Liangxiu Han, Xin Zhang, Haoming Chen, Daoqiang Zhang

View

Download (PDF)

Tags: Computer science, CUDA, Data parallelism, Deep learning, Neural networks, nVidia, Tesla V100

April 18, 2021 by hgpu

HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism

Jay H. Park, Gyeongchan Yun, Chang M. Yi, Nguyen T. Nguyen, Seungmin Lee, Jaesik Choi, Sam H. Noh, Young-ri Choi

View

Download (PDF)

Tags: Computer science, CUDA, Data parallelism, Deep learning, GPU cluster, Heterogeneous systems, Neural networks, nVidia, nVidia GeForce GTX Titan V, nVidia GeForce RTX 2060, nVidia Quadro P 4000, nVidia Titan RTX

May 31, 2020 by hgpu

The OoO VLIW JIT Compiler for GPU Inference

Paras Jain, Xiangxi Mo, Ajay Jain, Alexey Tumanov, Joseph E. Gonzalez, Ion Stoica

View

Download (PDF)

Tags: Compilers, Computer science, CUDA, Data parallelism, Machine learning, nVidia, Tesla V100

February 3, 2019 by hgpu

Over-synchronization in GPU Programs

profile_util: simple C++ library to report memory, timing, thread affinity

Profile Util library: A quick and easy way to get MPI, OpenMP and GPU runtime information

MaC: Mamba CPU performance predictor

MambaCPU: Enhanced Correlation Mining with State Space Models for CPU Performance Prediction

Mixed-precision finite element kernels

Mixed-precision finite element kernels and assembly: Rounding error analysis and hardware acceleration

SHMT: The Simultaneous and Heterogenous Multithreading project

General-Purpose Computing on Tensor Processors

Superpipeline: A Universal Approach for Reducing GPU Memory Usage in Large Models

EnergyUCB-Bandit

Online Energy Optimization in GPUs: A Multi-Armed Bandit Approach

See all packages

* * *

high performance computing on graphics processing units: hgpu.org

A Review of the Parallelization Strategies for Iterative Algorithms

Novel Parallelization Strategies for High-Performance DNN Training on HPC Systems

SYCL compute kernels for ExaHyPE

The Celerity High-level API: C++20 for Accelerator Clusters

A ML-based resource utilization OpenCL GPU-kernel fusion model

Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

GCN Inference Acceleration using High-Level Synthesis

Sylkan: Towards a Vulkan Compute Target Platform for SYCL

Efficient Large-Scale Language Model Training on GPU Clusters

A Hybrid Parallelization Approach for Distributed and Scalable Deep Learning

HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelism

The OoO VLIW JIT Compiler for GPU Inference

Recent source codes

Kokkidio: Kokkos+Eigen - write fast, readable multi-platform code

LLaMA-Inference-Bench

MATAR

Over-synchronization in GPU Programs

profile_util: simple C++ library to report memory, timing, thread affinity

MaC: Mamba CPU performance predictor

Mixed-precision finite element kernels

SHMT: The Simultaneous and Heterogenous Multithreading project

Superpipeline: A Universal Approach for Reducing GPU Memory Usage in Large Models

EnergyUCB-Bandit

Most viewed papers (last 30 days)