Immediate NeRF from webcams

[Ednaordinary]

This demo uses 12 webcams to continuously update the input frames

Setup:
https://github.com/NVlabs/instant-ngp/assets/88869424/68d02afa-8515-409e-ace3-247826083778

Demo 1:
https://github.com/NVlabs/instant-ngp/assets/88869424/37ad3cf0-237a-4d6c-9bdf-f1bb62c4468e

Demo 2:
https://github.com/NVlabs/instant-ngp/assets/88869424/86d32c35-e3af-4c2f-8574-5e9e62b7ca5b

Setup Details

Computer is equipped with Ryzen 7 5700X, RTX 3090 Ti, and 64 GB of ram The webcams are all VAVA 2K, I got them cheap for $100 for all Webcams are plugged into two different USB hubs, at any time 2 from each (4 total) can be accessed at 2592x1944 (Largest FOV) 30 fps I modified the script from the scripts/run.py file, a second thread runs that updates the views from all cameras once and writes them, then when that is complete the primary thread reloads them into testbed and relaunches the second thread to start updating again, so that there is little performance lost in the main thread. This method works best right after resetting training, then seems to loose the ability to modify the NeRF over time.

[Ednaordinary]

Script

Note that this script is **not meant to work out of the box**

https://gist.github.com/Ednaordinary/855dada2bd655bc135ce29eb93e4603d

Please modify the following lines:
83: Inside the list add the numbers for the cameras you want (to get this do v4l2-ctl --list-devices, then if you have /dev/video1 and want to use it, put 1)
84-89 and 101-104: These lines are to declare lists for multithreading. The simplest thing to do while still having a working script is to remove 84-89 and replace 101-104 with threads.append(threading.Thread(target=camloop, args=[cams, 0])), though note this will be slower. If you would like to multithread, make lists for each set of cams and make new threads.append(threading.Thread(target=camloop, args=[cams, 0])) lines, replacing cams with each list and adding an offset number of the length of the previous list of cams (so that they do not overwrite eachother). At any time 1 camera will be accessed from each list. All cams are not accessed at once because of bandwidth issues.
91: Add the directory for the scene and include "/images"
92: Change -framerate 30 -video_size 2592x1944 to the resolution and framerate for your camera

Once the script is ready calling it with zero arguments will capture a picture from each webcam and write it, then do nothing. This is useful for capturing pictures just once and then using colmap2nerf.py. Once you have transforms.json, run it with "--scene ./data/nerf/webcam --gui --train --n_steps 100000000000" replacing ./data/nerf/webcam with the directory to the scene. n_steps is the number of steps to train, but we don't want to stop training, so set it really high.

[michaelrubloff]

This is really sick!

[kafei123456]

Good, I will try this crazy idea!

[kafei123456]

hello, I wonder how your camera pos input? I tempt to use one camera to get multiple-view, but my "transforms.json" file is seemingly wrong. I don't know how transform world pos to instant_gpn pos.

[Ednaordinary]

In my implementation all that I did for position reconstruction was get pictures from each of the 12 webcams and use the colmap2nerf script to reconstruct. The cameras aren't actually positioned accurately enough for me to reconstruct the setup myself

[kafei123456]

This is a challenge!