Newton's method in optimization

Newton's method in optimization applies the Newton-Raphson iteration to the derivative of a twice-differentiable function in order to find its stationary points, which may be minima, maxima or saddle points. In one variable the method builds a sequence of points by fitting a quadratic approximation at the current point and moving to the point where that quadratic is minimal, which amounts to fitting a parabola with the same slope and curvature. If the function is itself quadratic, the exact extremum is found in a single step. In several variables the derivative is replaced by the gradient and the reciprocal of the second derivative by the inverse of the Hessian matrix. Because inverting the Hessian is costly in high dimensions, practical implementations solve a linear system instead, and step sizes other than one give the damped Newton method. Quasi-Newton methods approximate the Hessian from changes in the gradient.