3380

A Chunking Method for Euclidean Distance Matrix Calculation on Large Dataset Using Multi-GPU

Qi Li, V. Kecman, R. Salman
Dept. of Comput. Sci. Sch. of Eng., Virginia Commonwealth Univ., Richmond, VA, USA
Machine Learning and Applications (ICMLA), 2010 Ninth International Conference on

@conference{li2011chunking,

   title={A Chunking Method for Euclidean Distance Matrix Calculation on Large Dataset Using Multi-GPU},

   author={Li, Q. and Kecman, V. and Salman, R.},

   booktitle={Machine Learning and Applications (ICMLA), 2010 Ninth International Conference on},

   pages={208–213},

   year={2011},

   organization={IEEE}

}

Source Source   

2584

views

Calculating Euclidean distance matrix is a data intensive operation and becomes computationally prohibitive for large datasets. Recent development of Graphics Processing Units (GPUs) has produced superb performance on scientific computing problems using massive parallel processing cores. However, due to the limited size of device memory, many GPU based algorithms have low capability in solving problems with large datasets. In this paper, a chunking method is proposed to calculate Euclidean distance matrix on large datasets. This is not only designed for scalability in multi-GPU environment but also to maximize the computational capability of each individual GPU device. We first implement a fast GPU algorithm that is suitable for calculating sub matrices of Euclidean distance matrix. Then we utilize a Map-Reduce like framework to split the final distance matrix calculation into many small independent jobs of calculating partial distance matrices, which can be efficiently solved by our GPU algorithm. The framework also dynamically allocates GPU resources to those independent jobs for maximum performance. The experimental results have shown a speed up of 15x on datasets which contain more than half million data points.
No votes yet.
Please wait...

* * *

* * *

HGPU group © 2010-2024 hgpu.org

All rights belong to the respective authors

Contact us: