CMA-ES

Covariance matrix adaptation evolution strategy, or CMA-ES, is a stochastic, derivative-free strategy for numerical optimization of non-linear or non-convex continuous problems, belonging to the class of evolutionary algorithms. New candidate solutions are sampled from a multivariate normal distribution, and the covariance matrix adaptation step updates the covariance matrix of that distribution to learn a second-order model of the objective function, similar in spirit to approximating the inverse Hessian matrix in a quasi-Newton method, while requiring only a ranking of candidate solutions rather than derivatives or an explicit objective function. It is particularly useful when the underlying objective function is ill-conditioned.