CMA-ES for Hyperparameter Optimization of Deep Neural Networks
Univesity of Freiburg, Freiburg, Germany
arXiv:1604.07269 [cs.NE], (25 Apr 2016)
@article{loshchilov2016cmaes,
title={CMA-ES for Hyperparameter Optimization of Deep Neural Networks},
author={Loshchilov, Ilya and Hutter, Frank},
year={2016},
month={apr},
archivePrefix={"arXiv"},
primaryClass={cs.NE}
}
Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization. As an alternative, we propose to use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which is known for its state-of-the-art performance in derivative-free optimization. CMA-ES has some useful invariance properties and is friendly to parallel evaluations of solutions. We provide a toy example comparing CMA-ES and state-of-the-art Bayesian optimization algorithms for tuning the hyperparameters of a convolutional neural network for the MNIST dataset on 30 GPUs in parallel.
April 26, 2016 by hgpu
