Limited-memory BFGS

Limited-memory BFGS (L-BFGS or LM-BFGS) is an optimization algorithm in the family of quasi-Newton methods that approximates the Broyden–Fletcher–Goldfarb–Shanno algorithm (BFGS) using a limited amount of computer memory.^[1] It is a popular algorithm for parameter estimation in machine learning.^[2][3] The algorithm's target problem is to minimize ${\displaystyle f(\mathbf {x} )}$ over unconstrained values of the real-vector ${\displaystyle \mathbf {x} }$ where ${\displaystyle f}$ is a differentiable scalar function.

Like the original BFGS, L-BFGS uses an estimate of the inverse Hessian matrix to steer its search through variable space, but where BFGS stores a dense ${\displaystyle n\times n}$ approximation to the inverse Hessian (n being the number of variables in the problem), L-BFGS stores only a few vectors that represent the approximation implicitly. Due to its resulting linear memory requirement, the L-BFGS method is particularly well suited for optimization problems with many variables. Instead of the inverse Hessian H_k, L-BFGS maintains a history of the past m updates of the position x and gradient ∇f(x), where generally the history size m can be small (often ${\displaystyle m<10}$). These updates are used to implicitly do operations requiring the H_k-vector product.