Optimization (Wiki)

• Overview
  • An overview of gradient descent optimization algorithms (2017) Sebastian Ruder
  • Derivative tests

Linear programming

• Simplex algorithm (Wiki)

Gradient-based optimization

• Batch gradient descent
• Stochastic gradient descent
  • An Alternative View: When Does SGD Escape Local Minima? (2018) Robert Kleinberg, Yuanzhi Li, Yang Yuan
• Mini-batch gradient descent
• Momentum
• Nesterov accelerated gradient
• Adagrad
• Adadelta
• RMSprop
• Adam, AdaMax
  • Adam: A Method for Stochastic Optimization (2017) Diederik P. Kingma, Jimmy Ba
• TAdam
  • TAdam: A Robust Stochastic Gradient Optimizer (2020) Wendyam Eric Lionel Ilboudo, Taisuke Kobayashi, Kenji Sugimoto
• Nadam
• AMSGrad
• LaProp (Code)
  • LaProp: a Better Way to Combine Momentum with Adaptive Gradient (2020) Liu Ziyin, Zhikang T.Wang, Masahito Ueda

Bayesian optimization

• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning (2010) Eric Brochu, Vlad M. Cora, Nando de Freitas
• Practical Bayesian Optimization of Machine Learning Algorithms (2012) Jasper Snoek, Hugo Larochelle, Ryan P. Adams
• Taking the Human Out of the Loop: A Review of Bayesian Optimization (2016) Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, Nando de Freitas
• REMBO
  • Bayesian Optimization in High Dimensions via Random Embeddings (2013) Ziyu Wang, Masrour Zoghi, Frank Hutter, David Matheson, Nando De Freitas
  • On the choice of the low-dimensional domain for globaloptimization via random embedding (2018) Mickael Binois, David Ginsbourger, Olivier Roustant
• HeSBO
  • A Framework for Bayesian Optimization in Embedded Subspaces (2019) Alexander Munteanu, Amin Nayebi, Matthias Poloczek
• SIRBO
  • High Dimensional Bayesian Optimization via Supervised Dimension Reduction (2019) Miao Zhang, Huiqi Li, Steven Su
• SI-BO
• SILBO
  • Semi-supervised Embedding Learning for High-dimensionalBayesian Optimization (2020) Jingfan Chen, Guanghui Zhu, Rong Gu, Chunfeng Yuan, Yihua Huang
• BOHB - Bayesian Optimization with Hyperband
  • BOHB: Robust and Efficient Hyperparameter Optimization at Scale (2018) Stefan Falkner, Aaron Klein, Frank Hutter

Evolutionary algorithms (Wiki)

• Imitate some aspects of natural evolution
• GA Genetic algorithm (Wiki)
• MA Memetic algorithm (Wiki)
• GP Genetic programming
• ES Evolutionary strategies
• EP Evolutionary programming
• LCS Learning classifier systems
• Harmony search
• PBT Population-based Training
  • Population Based Training of Neural Networks (2017) Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, Chrisantha Fernando, Koray Kavukcuoglu

Trajectory-based algorithms

• Simulated annealing
  • Approximate Solution of Certain Types of Constrained Optimization Problems (1970) Martin Pincus
  • Optimization by Simulated Annealing (1983) S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi
• Hill climbing
• b-Hill climbing
• Tabu search
• Variable neighborhood search

Swarm-based algorithms

• Artificial bee colony
• Cuckoo search
• Firefly algorithm
• Particle swarm (Wiki)
  • Particle Swarm Optimization (1995) James Kennedy, Russell Eberhart

Multi-armed bandits (Wiki)

• Some Aspects of the Sequential Design of Experiments (1952) Herbert Robbins
• Multi-armed Bandit Algorithmsand Empirical Evaluation (2005) Joannes Vermorel, Mehryar Mohri
• A modern Bayesian look at the multi-armed bandit (2010) Steven L. Scott

Heuristic algorithms

Multimodel optimization

Multiobjective optimization

• Pareto set

Software

• Python

  • Sherpa (Docs, Code, Paper)
  • PyMOO (Homepage, Paper)

• C++

  • Emsmallen (Homepage)