Evaluation of a novel Momentum Based Quasi-Newton Method in Optimization Problems

This work proposes a novel application of momentum to the L-BFGS Quasi-Newton method and seeks to evaluate its performance when applied to optimization problems. We implement and document the proposed optimization algorithm and other algorithms that utilize the Quasi-Newton method without momentum in the ScalaTion framework with the aid of a C library implementation. We then compare the performance of these algorithms across different scenarios using a similar set of hyperparameters. The results presented indicate specific instances where combining momentum with the L-BFGS Quasi-Newton method leads to improvement in function optimization results. Future work seeks to expand on this initial evaluation by further testing the proposed DML-BFGS optimization algorithm and exploring other approaches that combine momentum with Quasi-Newton methods.