Spark-GPU: An Accelerated In-Memory Data Processing Engine on Clusters

Yuan Yuan, Meisam Fathi Salmi, Yin Huai, Kaibo Wang, Rubao Lee, Xiaodong Zhang
The Ohio State University
IEEE International Conference on Big Data (IEEE BigData),2016


   title={Spark-GPU: An Accelerated In-Memory Data Processing Engine on Clusters},

   author={Yuan, Yuan and Salmi, Meisam Fathi and Huai, Yin and Wang, Kaibo and Lee, Rubao and Zhang, Xiaodong},



Download Download (PDF)   View View   Source Source   



Apache Spark is an in-memory data processing system that supports both SQL queries and advanced analytics over large data sets. In this paper, we present our design and implementation of Spark-GPU that enables Spark to utilize GPU’s massively parallel processing ability to achieve both high performance and high throughput. Spark-GPU transforms a general-purpose data processing system into a GPU-supported system by addressing several real-world technical challenges including minimizing internal and external data transfers, preparing a suitable data format and a batching mode for efficient GPU execution, and determining the suitability of workloads for GPU with a task scheduling capability between CPU and GPU. We have comprehensively evaluated Spark-GPU with a set of representative analytical workloads to show its effectiveness. Our results show that Spark-GPU improves the performance of machine learning workloads by up to 16.13x and the performance of SQL queries by up to 4.83x.
Rating: 2.5/5. From 1 vote.
Please wait...

* * *

* * *

HGPU group © 2010-2024 hgpu.org

All rights belong to the respective authors

Contact us: