Spark-GPU: An Accelerated In-Memory Data Processing Engine on Clusters
The Ohio State University
IEEE International Conference on Big Data (IEEE BigData),2016
@article{yuan2016spark,
title={Spark-GPU: An Accelerated In-Memory Data Processing Engine on Clusters},
author={Yuan, Yuan and Salmi, Meisam Fathi and Huai, Yin and Wang, Kaibo and Lee, Rubao and Zhang, Xiaodong},
year={2016}
}
Apache Spark is an in-memory data processing system that supports both SQL queries and advanced analytics over large data sets. In this paper, we present our design and implementation of Spark-GPU that enables Spark to utilize GPU’s massively parallel processing ability to achieve both high performance and high throughput. Spark-GPU transforms a general-purpose data processing system into a GPU-supported system by addressing several real-world technical challenges including minimizing internal and external data transfers, preparing a suitable data format and a batching mode for efficient GPU execution, and determining the suitability of workloads for GPU with a task scheduling capability between CPU and GPU. We have comprehensively evaluated Spark-GPU with a set of representative analytical workloads to show its effectiveness. Our results show that Spark-GPU improves the performance of machine learning workloads by up to 16.13x and the performance of SQL queries by up to 4.83x.
February 5, 2017 by hgpu
