Skip to content

Stochastic Quasi-Newton Methods in a Trust Region Framework (MATLAB implementation)

Notifications You must be signed in to change notification settings

MATHinDL/sL_QN_TR

Repository files navigation

blue-lines-png-7

sL_QN_TR

Stochastic Quasi-Newton Methods in a Trust Region Framework (MATLAB Implementation)

The repository "sL_QN_TR" contains programs of s-LBFGS-TR and s-LSR1-TR which are stochastic algorithms based on two well-known quasi-Newton updates, i.e., limited memory BFGS and limited memory SR1, in a Trust Region Framework. These algorithms and their performance in training deep neural networks for image classification tasks are described in the following article:

Deep Neural Networks Training by Stochastic Quasi-Newton Trust-Region Methods

Mahsa Yousefi and Angeles Martinez Calomardo

Download and Read the Paper.

Read the supplementary material of the paper for more numerical results.

A MATLAB-based tutorial on implementing training loops for a deep neural network

Mahsa Yousefi and Angeles Martinez Calomardo

This tutorial shows how to define a convolutional neural network (CNN) and how to create and customize your training loops. If you are a MATLAB user who would like to implement your training algorithm for which the MATLAB built-in function does not exist, read this tutorial.

Download and Read the Tutorial.

blue-lines-png-7

In this repository,

  • all programs have been written in MATLAB using the DL toolbox.
  • there are four main programs as below:
    • sL_BFGS_TR.mlx
    • sL_SR1_TR.mlx
    • sL_BFGS_TR_noBN.mlx
    • sL_SR1_TR_noBN.mlx
  • every single program in .mlx format (live script of MATLAB) provides a step-by-step guideline for users.
  • the folder Subroutines includes required functions (.m files) for running programs.
  • the folder Datasets consists of three folders associated with standard benchmarks MNIST, Fashion-MNIST, and CIFAR10 with their loading file.
  • three considered main architectures with and without batch normalization layers are:
    • LeNet-like
    • ResNet-20
    • ConvNet3FC2
  • the networks without batch normalization layers are included in sL_BFGS_TR_noBN.mlx and sL_SR1_TR_noBN.mlx programs.

Remark: For executing, put together main programs with all .m files and required files from Subroutines and Datasets, respectively.

blue-lines-png-7

Please contact me if you have any questions, suggestions, requests, or bug reports.

[email protected]