Importance of Data Loading Pipeline in Training Deep Neural Networks
Huawei Noah’s Ark Lab, Montreal, QC H3N 1X9, Canada
arXiv:2005.02130 [cs.CV], (21 Apr 2020)
@misc{zolnouri2020importance,
title={Importance of Data Loading Pipeline in Training Deep Neural Networks},
author={Mahdi Zolnouri and Xinlin Li and Vahid Partovi Nia},
year={2020},
eprint={2005.02130},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Training large-scale deep neural networks is a long, time-consuming operation, often requiring many GPUs to accelerate. In large models, the time spent loading data takes a significant portion of model training time. As GPU servers are typically expensive, tricks that can save training time are valuable.Slow training is observed especially on real-world applications where exhaustive data augmentation operations are required. Data augmentation techniques include: padding, rotation, adding noise, down sampling, up sampling, etc. These additional operations increase the need to build an efficient data loading pipeline, and to explore existing tools to speed up training time. We focus on the comparison of two main tools designed for this task, namely binary data format to accelerate data reading, and NVIDIA DALI to accelerate data augmentation. Our study shows improvement on the order of 20% to 40% if such dedicated tools are used.
May 10, 2020 by hgpu
