Get real world coordinates from NGP scene

[alex-klimov]

Hi guys, excellent project!

What I'm trying to understand - is that possible to use trained NGP scene to guide robot inside a scene? Is that possible to retrieve real-world coordinates from the scene?

What would you recommend as the origin for real-world coordinates? I have a rig with multiple static cameras around the scene - can I map the origin to one of the cameras or known point on the rig?

Thanks,
Alex

[Tom94]

Hi Alex, the NeRF model is fit within a bounding box with side-length aabb_scale and central point [0.5, 0.5, 0.5]. The highest resolution reconstruction will happen around this central point.

So for best results, I recommend translating and scaling/translating your real-world coordinates such that they fit into the bounding box and convert back in post.

[alex-klimov]

Is there a deterministic relationship between the aabb bounding box position, the objects inside the box and the position of the cameras specified in transforms.json?

[Tom94]

Yes, the transformation is a bit convoluted to make the Y=UP coordinate system of NGP compatible with NeRF's datasets (90 degree rotation + scale factor), the implementation details of which you can find in nerf_loader.cu/.h. You can also specify custom offset and scale parameters in the json (again, see nerf_loader.cu/.h on how these are incorporated into the conversion). Cheers!

[alex-klimov]

Love it! Looking through the code.

[chl2]

Hello, I still don't know how to modify it. My current coordinate system is the GPS coordinate system. How can I normalize it? Why does the relative position of the cameras change when I operate according to the colmap2nerf.py?