NeRF

Implementing NeRF model from scratch

v1_coarse_fine
Chair	Ship	Hotdog

v11
Hotdog

v16
Lego	Hotdog	Materials

Final Version (v18) (Special thanks to: [4])

Custom Dataset

1. Capture a video (forward facing), place it in Image2Pose_NERF/customdataset folder.
2. Open terminal in that folder and extract frames using []:

   ffmpeg -i input.mp4 '%04d.png'

3. Create a folder name 'extract' inside customdataset and place the extracted image in this folder.
4. Reduce the number of frames using filterimage.py code present in Image2Pose_NERF/customdataset folder.
5. Open COLMAP, click on new project -> select the customdataset for database and customdataset/images for image path -> save
6. Now in COLMAP select Reconstruction option -> Auto Reconstruction -> Select workspace as customdataset -> Select image folder as customdataset/images -> select data type as video frames -> check shared intrinsics -> check sparse -> uncheck dense and now click Run. It will create Sparse name folder in customdataset.
7. Now, in Image2Pose_NERF folder run image2pose.py, it will create poses_bounds.npy file []:

   python image2pose.py customdataset/

8. customdataset folder is final dataset.

Hyperparameters Synthetic:

Parameters	Values
Iteration	200K
Scheduler	Exponential Decay
Scheduler Step	160K approx.
Rays Sample	1024
Crop	0.5
Pre Crop Iter	50
Factor	2
Near Plane	2.0
Far Plane	6.0
Height	800 / factor
Width	800 / factor
Downscale	2
lr	5e-4
lrsch_gamma	0.1
Pos Enc Dim	10
Dir Enc Dim	4
Num Samples	64
Num Samples Fine	128
Net Dim	256
Net Depth	8
Inp Feat	2*(num_channels*pos_enc_dim) + num_channels
Dir Feat	2*(num_channels*dir_enc_dim) + num_channels

References:

[1] Computer Graphics and Deep Learning with NeRF using TensorFlow and Keras []

[2] 3D volumetric rendering with NeRF []

[3] Nerf Official Colab Notebook []

[4] NeRF PyTorch [] ( Special Thanks :) )

[5] PyTorch Image Quality (PIQ) []