Second-order optimisation?

[GilesStrong]

Idea

Since we're dealing with comparatively few parameters (4 layers * 10*10 voxels * 2 parameters per voxel = 800 params), we may want to investigate whether optimisation based on the (approximation) of the Hessian of the loss provides quicker convergence. PyTorch includes L-BFGS, but it only works for single parameter groups, and Newtonian optimisation should be implementable.

Possible problems

• Memory requirements
  • L-BFGS has a history size which can be reduced
  • An 800x800 Hessian should be fine, though, I'd have thought
• Learning rate scale
  • Currently we use two different learning rates due to the different scales of the resolution and efficiency. Probably we could keep the parameters at the same scale (N(0,1)) and then rescale them during application in the forward pass.