17533

Accelerating recurrent neural network training using sequence bucketing and multi-GPU data parallelization

Viacheslav Khomenko, Oleg Shyshkov, Olga Radyvonenko, Kostiantyn Bokhan
Samsung RnD Institute Ukraine (SRK), 57, L’va Tolstogo Str., Kyiv, 01032, Ukraine
arXiv:1708.05604 [cs.LG], (18 Aug 2017)

@article{khomenko2017accelerating,

   title={Accelerating recurrent neural network training using sequence bucketing and multi-GPU data parallelization},

   author={Khomenko, Viacheslav},

   year={2017},

   month={aug},

   archivePrefix={"arXiv"},

   primaryClass={cs.LG},

   doi={10.1109/DSMP.2016.7583516}

}

Download Download (PDF)   View View   Source Source   

1854

views

An efficient algorithm for recurrent neural network training is presented. The approach increases the training speed for tasks where a length of the input sequence may vary significantly. The proposed approach is based on the optimal batch bucketing by input sequence length and data parallelization on multiple graphical processing units. The baseline training performance without sequence bucketing is compared with the proposed solution for a different number of buckets. An example is given for the online handwriting recognition task using an LSTM recurrent neural network. The evaluation is performed in terms of the wall clock time, number of epochs, and validation loss value.
Rating: 1.0/5. From 1 vote.
Please wait...

* * *

* * *

Featured events

2018
November
27-30
Hida Takayama, Japan

The Third International Workshop on GPU Computing and AI (GCA), 2018

2018
September
19-21
Nagoya University, Japan

The 5th International Conference on Power and Energy Systems Engineering (CPESE), 2018

2018
September
22-24
MediaCityUK, Salford Quays, Greater Manchester, England

The 10th International Conference on Information Management and Engineering (ICIME), 2018

2018
August
21-23
No. 1037, Luoyu Road, Hongshan District, Wuhan, China

The 4th International Conference on Control Science and Systems Engineering (ICCSSE), 2018

2018
October
29-31
Nanyang Executive Centre in Nanyang Technological University, Singapore

The 2018 International Conference on Cloud Computing and Internet of Things (CCIOT’18), 2018

HGPU group © 2010-2018 hgpu.org

All rights belong to the respective authors

Contact us: