Towards High-Performance and Cost-Effective Distributed Storage Systems with Information Dispersal Algorithms

Dongfang Zhao, Kent Burlingame, Corentin Debains, Pedro Alvarez-Tabio, Ioan Raicu
Illinois Institute of Technology, Chicago, IL, USA
IEEE Cluster, 2013


   title={Towards High-Performance and Cost-Effective Distributed Storage Systems with Information Dispersal Algorithms},

   author={Zhao, Dongfang and Burlingame, Kent and Debains, Corentin and Alvarez-Tabio, Pedro and Raicu, Ioan},

   booktitle={IEEE International Conference on Cluster Computing, IEEE CLUSTER},




Download Download (PDF)   View View   Source Source   



Reliability is one of the most fundamental challenges for high performance computing (HPC) and cloud computing. Data replication is the de facto mechanism to achieve high reliability, even though it has been criticized for its high cost and low efficiency. Recent research showed promising results by switching the traditional data replication to a software-based RAID. In order to systematically study the effectiveness of this new method, we built two storage systems from the ground up: a POSIX-compliant distributed file system (FusionFS) and a distributed key-value store (IStore), both supporting information dispersal algorithms (IDA) for data redundancy. FusionFS is crafted to have excellent throughput and scalability for HPC, whereas IStore is architected mainly as a light-weight key-value storage in cloud computing. We evaluated both systems with a large number of parameter combinations. Results show that, for both HPC and cloud computing communities, IDA-based methods with current commodity hardware could outperform data replication in some cases, and would completely surpass data replication with the growing computational capacity through multi/many-core processors (e.g. Intel Xeon Phi, NVIDIA GPU).
No votes yet.
Please wait...

* * *

* * *

HGPU group © 2010-2021 hgpu.org

All rights belong to the respective authors

Contact us: