Confusing coordinate system.

[mcourteaux]

I'm trying to wrap my head around the coordinate system, and it looks unlike anything I've seen before. I setup a simple test transforms.json:

{
    "camera_angle_x": 2.1273956954128375,
    "scale": 1.0,
    "offset": [
        0.0,
        0.0,
        0.0
    ],
    "frames": [
        {
            "file_path": "png/view_f0000f",
            "transform_matrix": [
                [
                    1.0,
                    0.0,
                    0.0,
                    0.8
                ],
                [
                    0.0,
                    1.0,
                    0.0,
                    1.0
                ],
                [
                    0.0,
                    0.0,
                    1.0,
                    1.2
                ],
                [
                    0.0,
                    0.0,
                    0.0,
                    1.0
                ]
            ]
        }
    ]
}

Notice how the camera I set up is having identity rotational part, and translation vector [0.8, 1.0, 1.2]. The result looks like this:

Confusing things:

• The camera and the unit cube have different orientations for X, Y, and Z (i.e.: red, green, blue axis). (see first image)
• The position of the camera relative to the unit cube seems wrong. The the camera seems to be at x=1, and z=0.8, whereas I specified x to be 0.8, and z to be 1.2. (see second image).
• The camera is at y=1.2, whereas it should be 1.0. (see third image).

So, position in the testbed is (1, 1.2, 0.8), but it should be (0.8, 1.0, 1.2). This shuffles around ALL axis. After long searching, I found this snippet:

instant-ngp/include/neural-graphics-primitives/nerf_loader.h

Lines 50 to 70 in 409613a

	auto nerf_matrix_to_ngp(const Eigen::Matrix<float, 3, 4>& nerf_matrix) {
	Eigen::Matrix<float, 3, 4> result;
	int X=0,Y=1,Z=2;
	result.col(0) = Eigen::Vector3f{ nerf_matrix(X,0), nerf_matrix(Y,0), nerf_matrix(Z,0)};
	result.col(1) = Eigen::Vector3f{-nerf_matrix(X,1), -nerf_matrix(Y,1), -nerf_matrix(Z,1)};
	result.col(2) = Eigen::Vector3f{-nerf_matrix(X,2), -nerf_matrix(Y,2), -nerf_matrix(Z,2)};
	result.col(3) = Eigen::Vector3f{ nerf_matrix(X,3), nerf_matrix(Y,3), nerf_matrix(Z,3)} * scale + offset;
	if (from_mitsuba) {
	result.col(0) *= -1;
	result.col(2) *= -1;
	} else {
	// Cycle axes xyz->yzx
	Eigen::Vector4f tmp = result.row(0);
	result.row(0) = (Eigen::Vector4f)result.row(1);
	result.row(1) = (Eigen::Vector4f)result.row(2);
	result.row(2) = tmp;
	}
	return result;
	}

So, my question kinda reduces to: how to think about this coordinate system, as I still haven't figured out how to convert my dataset transformations to yours.

[mmalex]

sorry for the shift in coordinate system! internally, ngp uses entirely 0-1 bounding box, as displayed in the gui, with cameras looking down positive z. that's just the convention we chose early on. however, we wanted to be compatible specifically with the original nerf datasets, which place the origin at 0, scale the cameras to be around 3 units from the origin, and have a different convention for 'up', camera up, etc. so the snippet you found could be described as 'converting from the original nerf paper conventions to ngp's conventions'. over time, the 'original nerf' format of transforms.json became the dominant (only practical) way to get data into ngp; so, it comes across as confusing as we don't explicitly talk about the mapping. In the end, I am afraid it is what it is. you can adjust the scale and offset components of the transformation by adding extra parameters to the json, see nerf_loader.cu that looks for keys 'offset', 'scale' and 'aabb'. however the coordinate flipping is hard coded for the time being.

…

On Fri, Jan 21, 2022 at 5:07 PM Martijn Courteaux ***@***.***> wrote: I'm trying to wrap my head around the coordinate system, and it looks unlike anything I've seen before. I setup a simple test transforms.json: { "camera_angle_x": 2.1273956954128375, "scale": 1.0, "offset": [ 0.0, 0.0, 0.0 ], "frames": [ { "file_path": "png/view_f0000f", "transform_matrix": [ [ 1.0, 0.0, 0.0, 0.8 ], [ 0.0, 1.0, 0.0, 1.0 ], [ 0.0, 0.0, 1.0, 1.2 ], [ 0.0, 0.0, 0.0, 1.0 ] ] } ] } Notice how the camera I set up is having identity rotational part, and translation vector [0.8, 1.0, 1.2]. The result looks like this: [image: image] <https://user-images.githubusercontent.com/845012/150566804-e932cdc4-b977-4a0c-9d4f-3ecaae490d93.png> [image: image] <https://user-images.githubusercontent.com/845012/150567363-a5ecf787-111a-46ca-9db7-1b499c115acd.png> [image: image] <https://user-images.githubusercontent.com/845012/150567796-e4dec845-4aea-463e-b456-751db1adf200.png> Confusing things: - The camera and the unit cube have different orientations for X, Y, and Z (i.e.: red, green, blue axis). (see first image) - The position of the camera relative to the unit cube seems wrong. The the camera seems to be at x=1, and z=0.8, whereas I specified x to be 0.8, and z to be 1.2. (see second image). - The camera is at y=1.2, whereas it should be 1.0. (see third image). So, position in the testbed is (1, 1.2, 0.8), but it should be (0.8, 1.0, 1.2). This shuffles around ALL axis. After long searching, I found this snippet: https://github.com/NVlabs/instant-ngp/blob/409613afdc08f69342a9269b9e674604229d183f/include/neural-graphics-primitives/nerf_loader.h#L50-L70 So, my question kinda reduces to: how to think about this coordinate system, as I still haven't figured out how to convert my dataset transformations to yours. — Reply to this email directly, view it on GitHub <#72>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFOYTZYYMZ3QX7H22NUVDTUXGHEHANCNFSM5MQA7I5Q> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[mcourteaux]

Looking at the fox dataset, it seems that the -y is the camera up axis. I'm still puzzling, haha...

(a few minutes later...)

This was key!! The camera orientation was flipped (y-down). Adding that 180 degree rotation upon converting makes it work.

Now, still rotate everything into the normal upward rotation in your coordinate system.

I looked through the code, and my intuition is that the up setting doesn't do anything for nerf.

[Yanall-Boutros]

i remember feeling infuriated when i found out unreal engine uses a left handed coordinate system.

[mcourteaux]

For reference for others. This is what I have right now, based on a position and euler rotations for a Blender-like camera:

def generate_transform_matrix(pos, rot):
    def Rx(theta):
      return np.matrix([[ 1, 0            , 0            ],
                        [ 0, np.cos(theta),-np.sin(theta)],
                        [ 0, np.sin(theta), np.cos(theta)]])
    def Ry(theta):
      return np.matrix([[ np.cos(theta), 0, np.sin(theta)],
                        [ 0            , 1, 0            ],
                        [-np.sin(theta), 0, np.cos(theta)]])
    def Rz(theta):
      return np.matrix([[ np.cos(theta), -np.sin(theta), 0 ],
                        [ np.sin(theta), np.cos(theta) , 0 ],
                        [ 0            , 0             , 1 ]])

    R = Rz(rot[2]) * Ry(rot[1]) * Rx(rot[0])
    xf_rot = np.eye(4)
    xf_rot[:3,:3] = R

    xf_pos = np.eye(4)
    xf_pos[:3,3] = pos - average_position

    # barbershop_mirros_hd_dense:
    # - camera plane is y+z plane, meaning: constant x-values
    # - cameras look to +x

    # Don't ask me...
    extra_xf = np.matrix([
        [-1, 0, 0, 0],
        [ 0, 0, 1, 0],
        [ 0, 1, 0, 0],
        [ 0, 0, 0, 1]])
    # NerF will cycle forward, so lets cycle backward.
    shift_coords = np.matrix([
        [0, 0, 1, 0],
        [1, 0, 0, 0],
        [0, 1, 0, 0],
        [0, 0, 0, 1]])
    xf = shift_coords @ extra_xf @ xf_pos
    assert np.abs(np.linalg.det(xf) - 1.0) < 1e-4
    xf = xf @ xf_rot
    return xf

[FlyFish-space]

What is the relationship between the coordinate axis before the conversion and the coordinate axis after your conversion?
(x,y,z -> y -x z ?)

[mcourteaux]

Well, I quickly ran the code, and the combined matrix that transforms the axis is this one:

M = 
[[ 0  1  0  0]
 [-1  0  0  0]
 [ 0  0  1  0]
 [ 0  0  0  1]]

When using this as: ngp = M * blender, I read this as: (x, y, z)_ngp = (y, -x, z)_blender. So yes, what you had seems about right. Note that just applying this axis reshuffling doesn't suffice, as it also rotates.

[mcourteaux]

I think this transforms too much. It seems to rotate 180° around the up (y-axis in NeRF) too much. I'm getting upset again.

[AndresPiola]

average_position

wha is average_position?

[mcourteaux]

average_position

wha is average_position?

For me that was the mean position of all cameras, which I used to subtract from all camera positions to get the scene to be close to the center of the bounding volume of the NeRF.

[mcourteaux]

Absolutely beautiful!!

nerf_barbershop_spherical_2.small.mp4

[mcourteaux]

For NeRF, you need to put the Minimum Distance to zero in the nerf training options. Once you do that, you might want to reset training.

[qhdqhd]

Thank you, may I trouble you to elaborate a little bit? How is Minimum Distance put to zero?

[mcourteaux]

It's the "Near Distance", under the "NeRF Training Options":
https://github.com/NVlabs/instant-ngp/blob/master/src/testbed.cu#L399

[qhdqhd]

Thanks a lot for your advice, it's very useful. I got the same result as you.

But I still found some problems:

1. The 256x256 image looks good, but it does lack some texture detail.

Does it mean that a larger number of small resolution images and a small number of large resolution images can achieve the same effect? I wonder why are you using such a small resolution?
2. The cameras are distributed on the surface of a sphere or cube. When the virtual viewpoint moves within the sphere or cube, the rendering result is good, but when the virtual viewpoint deviates from the training viewpoint, the rendering result becomes worse.

I would like to ask, why distribute the cameras on a sphere or a cube? Evenly distributing multiple viewpoints in the scene should give a freer virtual viewpoint.

[mcourteaux]

Amazing! I'm glad other people managed to recreate the results!

1. I chose those small textures as a quick test to not have to select a subset of images from the full dataset. But yes, generally, due to the aliasing, the results can be really good from very small aliased pictures, but a lot of them.
2. My field of research is light fields where data comes from a more regularly spaced camera setup. In the paper, I elaborate in greater detail what exactly I'm looking for that I provide in this dataset.

[mmalex]

thankyou for this! I am tempted to make a pullrequest for you that disables as much of the coordinate transformation as possible - dont worry the default wont change so your dataset will continue to work - I feel guilty about the trouble we caused :)

[mcourteaux]

I think I wouldn't be the only one that would appreciate a non-weirdly-behaving coordinate system 😋 I must admit, I got to this point by brute forcing transformation matrices and a little bit of iterative educated guessing. I'm confused as the camera upside-down thing we discussed earlier is not clearly in my transformation code anymore.

https://twitter.com/ID_AA_Carmack/status/432298265254449152

[jc211]

To remove transform nastiness from the nerf code, the only areas I found that need to be changed are in nerf_loader.h

Eigen::Vector3f nerf_direction_to_ngp(const Eigen::Vector3f& nerf_dir) {
  Eigen::Vector3f result = nerf_dir;
  return result;
}

Eigen::Matrix<float, 3, 4> nerf_matrix_to_ngp(const Eigen::Matrix<float, 3, 4>& nerf_matrix) {
  Eigen::Matrix<float, 3, 4> result = nerf_matrix;
  result.col(3) = result.col(3) * scale + offset;
  return result;
}

Eigen::Matrix<float, 3, 4> ngp_matrix_to_nerf(const Eigen::Matrix<float, 3, 4>& ngp_matrix) {
  Eigen::Matrix<float, 3, 4> result = ngp_matrix;
  result.col(3) = (result.col(3) - offset) / scale;
  return result;
}

void nerf_ray_to_ngp(Ray& ray, bool scale_direction = false) {
  ray.o = ray.o * scale + offset;
  if (scale_direction)
	  ray.d *= scale;
}

If you change this and compile, the transforms are expected to be in the opencv convention (and in this case the ngp convention) which is

Otherwise, the cameras in transforms.json are actually already in the blender convention (camera looking down -z and up is +y). You just have to make sure the intrinsic matrix is all positive.

[milliondabeat]

hey does anyone know how i could import a camera i tracked from a 3d software like blender into instant ngp as a camera path?

[mcourteaux]

You generate a path JSON file. To see an example, just play with the path editor in the testbed; export the path, and look at the file format. It's simple, but you'll need quaternions. Note that this does NOT do all of this transformation garbage; this just uses regular coordinates like how the internal engine works. Just keep in mind that positive-Y axis of the camera points down, just like @jc211 mentioned above.

[allAxisM]

'To see an example, just play with the path editor in the testbed; export the path'.

Any chance you could explain how to export the path? I can't seem to work this out. Also, how would you import a JSON file with a camera path?

Thanks :)

[TheLordSaiyan]

I think you guys could help me with my issue, I can't find the problem. The coordinates of the camera are off. #1286

[SergioMOrozco]

Hello, I am dealing with an issue that I feel is related. If anybody could help me, I would greatly appreciate it!

You can find my issue here #1360

[WillowProduction]

Hello every one,
Is there anyone who tried to import the saved preset camera path into Blender? the JSON file is really simple and works with kwyframes, but the interpolation inside ngp is so strange. Can't model this behaviour inside Blender