CMA-ES for Hyperparameter Optimization of Deep Neural Networks

Ilya Loshchilov, Frank Hutter
Univesity of Freiburg, Freiburg, Germany
arXiv:1604.07269 [cs.NE], (25 Apr 2016)


   title={CMA-ES for Hyperparameter Optimization of Deep Neural Networks},

   author={Loshchilov, Ilya and Hutter, Frank},






Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization. As an alternative, we propose to use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which is known for its state-of-the-art performance in derivative-free optimization. CMA-ES has some useful invariance properties and is friendly to parallel evaluations of solutions. We provide a toy example comparing CMA-ES and state-of-the-art Bayesian optimization algorithms for tuning the hyperparameters of a convolutional neural network for the MNIST dataset on 30 GPUs in parallel.
Rating: 2.5/5. From 4 votes.
Please wait...

* * *

* * *

HGPU group © 2010-2020 hgpu.org

All rights belong to the respective authors

Contact us: