one reconstructed example using own dataset(with black ghost around the recostructed people)

[neilgogogo]

I tried to input the cmu panoptic data into the data loader and find it is quite wired that there are several reconstructed humans in the scene, meanwhile, the human is lying instead of standing. I guess it might be the reason that the scale of T in extri is not correct? Has anyone encountered this problem?

Many thanks

[neilgogogo]

I find in my dataset, the camera coordinate is y right x up, while the fox dataset is y down, so maybe I need to rotate the matrix to y down and see what will happen

[neilgogogo]

1645691461455317.mp4

After the rotation of the camera coordinate system, the camera's extrinsic parameters seem to be OK, and the reconstructed object converges to one single object. but there is a lot of black fog in the reconstruction results. My training data is a picture with a black background. When I zoom into the box, I could see the reconstructed human.

[neilgogogo]

abce0acea2604baa2312805623081e9e_667384797948_v_1645691997958281.mp4

here is the zoom-in result, then I find I could filter out the black fog by using cropping AABB max-min. Although there are still some ghost which might be the reason of poor extri and intri parameters.

[qhdqhd]

How did you calibrate the cameras? using colmap or grundtruth? As far as I know, CMU data have different intrinsic parameters for different cameras, which is not suitable for NeRF.

[qhdqhd]

By the way, You said "filter out the black fog by using cropping AABB max-min", can you explain it ? I meet the same problem.

[neilgogogo]

I used the original image later, without masking out the background. And I think the result is better than using the masked image, which has more black area.

But you could also use the AABB max-min instead to filter out the unwanted black ghost, it is just like changing the rendering box and everything outside the box will not be rendered.

[qhdqhd]

I get it! Thank you. Generally, masking out the background really works.

[qhdqhd]

CMU data has 31 HD cameras, which is not enough for NeRF. So the black fog may relate to this?

[neilgogogo]

Yes, that might be the reason. I remember in the paper, it said 50-150 training views for Nerf.