Limited-Memory Broyden-Fletcher-Goldfarb-Shanno Algorithm in ML.NET
A short theoretical introduction to the L-BFGS algorithm and its implementation using ML.NET.
Some time ago I published an article about the implementation of Naive Bayes using ML.NET. Continuing this series today I would like to introduce you to the Limited-Memory Broyden-Fletcher-Goldfarb-Shanno method. I will start with a theory and an explanation of what this method is about and what it is used for.
Some facts
The creator of the algorithm is Jorge Nocedal. It was created at the Optimization Center, a joint venture of Argonne National Laboratory and Northwestern University. The original source code was written in FORTRAN. It is referred to as software for large-scale unconstrained optimization. As you might expect from the name, this method is similar to BFGS, but it uses less memory. Therefore, it is perfect for large datasets.
L-BFGS
This is an algorithm from the Quasi-Newton family of methods. These are algorithms for finding local extrema of functions, which are based on Newton’s method of finding stationary points of functions.
Time for some math