Support type-2 Fisher in KFAC

[runame]

As opposed to using MC samples from the model's predictive distribution.

[f-dangel]

Question: Is this already implemented in curvlinops.GGNLinearOperator?

[runame]

I don't think so, because curvlinops.GGNLinearOperator computes the exact GGN/Fisher, whereas this issue request a KFAC approximation using one backward pass per output dimension (see the implementation in ASDL as an example) to compute the gradients. I realize that the naming of the issue is a bit confusing, since it says "exact Fisher/GGN".

[f-dangel]

Oh okay, so this is sampling gradients versus using the columns of the loss-output Hessian's matrix square root? (I think in BackPACK this corresponds to KFLR)

[runame]

Yeah, that sounds correct.

[f-dangel]

I was trying to find a meaningful title for this. I could not find it on the internet, but believe the gradient-sampling-based estimation of the Fisher is called 'type-1', whereas the loss-Hessian-based estimation of the Fisher is called 'type-2'. Maybe @yorkerlin can help us out on this one.