Add a diagnostic to warn users if the GP training learns a 0 whitekernel

[odunbar]

It is not uncommon to see output from GP training that looks like this:

This comes from a 2D -> 2D problem training ARD kernel with GaussianProcess(...,noise_learn=true), (the default)

Type: GaussianProcesses.SumKernel{GaussianProcesses.SEArd{Float64}, GaussianProcesses.Noise{Float64}}
  Type: GaussianProcesses.SEArd{Float64}, Params: [-0.0, -0.0, 0.0]  Type: GaussianProcesses.Noise{Float64}, Params: [0.0]
created GP: 1
kernel in GaussianProcess:
Type: GaussianProcesses.SumKernel{GaussianProcesses.SEArd{Float64}, GaussianProcesses.Noise{Float64}}
  Type: GaussianProcesses.SEArd{Float64}, Params: [-0.0, -0.0, 0.0]  Type: GaussianProcesses.Noise{Float64}, Params: [0.0]
created GP: 2
PosDefException(2)
PosDefException(6)
optimized hyperparameters of GP: 1
Type: GaussianProcesses.SumKernel{GaussianProcesses.SEArd{Float64}, GaussianProcesses.Noise{Float64}}
  Type: GaussianProcesses.SEArd{Float64}, Params: [1.5898189312537052, 4.284595166329162, 3.31913265553596]  Type: GaussianProcesses.Noise{Float64}, Params: [-25.115891356746516]
PosDefException(5)
optimized hyperparameters of GP: 2
Type: GaussianProcesses.SumKernel{GaussianProcesses.SEArd{Float64}, GaussianProcesses.Noise{Float64}}
  Type: GaussianProcesses.SEArd{Float64}, Params: [8.94388355520155, 4.631539162650433, 6.172770146416866]  Type: GaussianProcesses.Noise{Float64}, Params: [-17.73352141936848]

Paying attention to the Noise kernel, it learns values -25 and -17 after training, (these are in log space) so the kernel learns a noise of exp(-25) and exp(-3) is learnt . The default regularization alg_reg_noise (=minimum noise) is 10^-3, so basically we end up with a noise of 10^-3.

Solution

• if the solution produces overly confident (spiked) posteriors: increase alg_reg_noise with noise_learn = true can help regularize the training more
• if the posterior with this result still looks reasonable: then alg_reg_noise may be too large, (i.e. the true noise you are trying to learn is less than alg_reg_noise)
• set noise_learn = false (which sets alg_reg_noise = 1 (in a normalized space) and doesn't learn the extra kernel) tends to solve the problem too

Action to be taken

Add a warning to users if the noise learnt << alg_reg_noise. Suggest the above solutions.