Hyper parameters to reproduce results

[martinbyrod]

Hi

This is very nice work. I'm trying to bake my own implementation of this and trying to reproduce the results. I'm doing it in pytorch so a lot of the speed is ruined but that is fine for now. I'm getting pretty good results but not quite as good as in the paper (I'm getting PSNR ~30 on Lego). There might be some bug in there but I'm also not sure I'm using the right hyper parameter settings as not everything is mentioned in the paper. Do you have some suggestions for how to figure out the details? I tried sifting through your code, but it was not super easy to find the right bits. I'll give this another go later though.

I'm looking for things like batch size, learning rate, etc. But an exact formula for network output to rgb would be great too. For instance the paper mentions you use "log space density" but it is not 100% clear to me what this means and there is no equation that defines it. Just knowing what parts are identical to the original NeRF paper would help too.

Thanks in advance and if you don't have time to answer all my questions that's fine, but just knowing learning rate and batch size would feel great so I can stop constantly fiddling with these 😏.

[Tom94]

Hi there, for learning rate / batch size info, see the latest version of the paper. These extra details got added a couple of days ago. :)

As for log-space density: it means that the volume density (a.k.a. sigma_t) is defined as exp(network_prediction). You can look for the network_to_density and network_to_rgb functions in the code to see exactly how the raw outputs of the neural networks are used. The equation for obtaining the pixel color from density and RGB predictions along a raymarch is identical to NeRF.

Cheers!

[martinbyrod]

Wow thanks a lot for the super quick answer! I'll check it out right away. Since I have you here though, just one more followup question. In the original paper they have alpha = 1 - exp(sigma * d). When you say sigma = exp(network_prediction), does that then mean that you get alpha = 1 - exp(exp(network_prediction) * d) then? Or do you define sigma differently than they? That's the bit that confused me.

[Tom94]

Yes, up to a missing minus sign: alpha = 1 - exp(-exp(network_prediction) * dt)

[Tom94]

I agree that this equation in isolation feels a little unintuitive.

A more intuitive view is that the visual effect of density depends more on its relative magnitude than on its absolute value: small fluctuations in density have a big effect in a thin fog where the baseline density is near-zero. But the same small fluctuations would have negligible effect when the density is large, e.g. behind a hard surface. Hence, it makes sense to let the network predict the order of magnitude of density (log-space) rather than predicting it at an absolute scale.

[martinbyrod]

🤦 minus sign... Anyway that's perfect! The exp-exp thing looked too strange so I assumed my interpretation was wrong. But I'll give that a spin then.

[mkim-MASI]

Are you able to pull out the density function (i.e. the parameters of the network) after training?