Medium: Optimization Algorithms and their Role in Machine Learning: A Quick Look at CMA-ES and BFGS

Using Perturbation to Optimize BFGS Algorithm

Using the same random start points for 30 dimensions to test CEC2013 benchmark functions to see if the end point perturbated by hypercubes in each iteration (replacing the end point with the point within the search space of the hypercube: the distance between the end point and the start point of each iteration *square root of 2 / 30 for each dimension) betters the standard BFGS.

To have a fair comparison, the experiment is under these conditions for both standard BFGS and BFGS with perturbation:
‼️ Control the random start point to be the same for both standard BFGS and BFGS with perturbation
‼️ If the new generated end point is too close to original end point, then generate a new point until it's not too close.
‼️ Min value of the loss function with perturbation VS Min value of the loss function w/o perturbaion
‼️ Max. evaluations: 300,000
‼️ Check if all the solutions generated in each evaluation are in the right search space.

Current Result (Jan. 2023)

When the perturbation bigger than 1% of bounds: