Hyperparameter optimization

In machine learning, hyperparameter optimization[1] or tuning is the problem of choosing a set of optimal hyperparameters for a learning algorithm. A hyperparameter is a parameter whose value is used to control the learning process, which must be configured before the process starts.[2]

Hyperparameter optimization determines the set of hyperparameters that yields an optimal model which minimizes a predefined loss function on a given data set.[3] The objective function takes a set of hyperparameters and returns the associated loss.[3] Cross-validation is often used to estimate this generalization performance, and therefore choose the set of values for hyperparameters that maximize it.[4]

  1. ^ Matthias Feurer and Frank Hutter. Hyperparameter optimization. In: AutoML: Methods, Systems, Challenges, pages 3–38.
  2. ^ Yang, Li (2020). "On hyperparameter optimization of machine learning algorithms: Theory and practice". Neurocomputing. 415: 295–316. arXiv:2007.15745. doi:10.1016/j.neucom.2020.07.061.
  3. ^ a b Claesen, Marc; Bart De Moor (2015). "Hyperparameter Search in Machine Learning". arXiv:1502.02127 [cs.LG].
  4. ^ Bergstra, James; Bengio, Yoshua (2012). "Random Search for Hyper-Parameter Optimization" (PDF). Journal of Machine Learning Research. 13: 281–305.