Limited-Memory BFGS Algorithm

Limited-memory BFGS, or L-BFGS, is an optimization algorithm among the quasi-Newton methods that approximates the Broyden-Fletcher-Goldfarb-Shanno algorithm using a limited amount of computer memory, and it is a popular choice for parameter estimation in machine learning. Like the original BFGS, it uses an estimate of the inverse Hessian matrix to steer its search through variable space, but where BFGS stores a dense matrix approximating the inverse Hessian, L-BFGS instead stores only a small history of past position and gradient updates that represent the approximation implicitly, giving it a linear memory requirement that suits optimization problems with many variables.