Skip to content

Latest commit

 

History

History
executable file
·
75 lines (58 loc) · 8.2 KB

README.md

File metadata and controls

executable file
·
75 lines (58 loc) · 8.2 KB
Raw

Hands-On NeRF with KAN

KAN: Kolmogorov-Arnold Networks is a promising challenger to traditional MLPs. We're thrilled about integrating KAN into NeRF! Is KAN suited for view synthesis tasks? What challenges will we face? How will we tackle them? We provide our initial observations and future discussion!

main.mp4

Update News

  • Release the implementation of KANeRF based on PyKAN.
  • 🔥🔥🔥 Release an efficient implementation of KANeRF based on Efficient-KAN, achiving 15x speedup! 🎉
  • 🎉🎉🎉 We will conduct more evaluation, stay tuned! : D

Installation

KANeRF is buid based on nerfstudio and Efficient-KAN. Please refer to the website for detailed installation instructions if you encounter any problems.

# create python env
conda create --name nerfstudio -y python=3.8
conda activate nerfstudio
python -m pip install --upgrade pip

# install torch
pip install torch==2.1.2+cu118 torchvision==0.16.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118
conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit

# install tinycudann
pip install ninja git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch

# install nerfstudio
pip install nerfstudio

# install efficient-kan
pip install git+https://github.com/Blealtan/efficient-kan.git

Performance Comparision

We integrate KAN and NeRFacto and compare KANeRF with NeRFacto in terms of model parameters, training time, novel view synthesis performance, etc. on the Blender dataset. Under the same network settings, KAN slightly outperforms MLP in novel view synthesis, suggesting that KAN possesses a more powerful fitting capability. However, KAN's inference and training speed are significantly** slower than those of MLP. Furthermore, with a comparable number of parameters, KAN underperforms MLP.

Model NeRFacto NeRFacto Tiny KANeRF
Trainable Network Parameters 8192 2176 7131
Total Network Parameters 8192 2176 10683
hidden_dim 64 8 8
hidden dim color 64 8 8
num layers 2 1 1
num layers color 2 1 1
geo feat dim 15 7 7
appearance embed dim 32 8 8
Training Time 14m 13s 13m 47s 37m 20s
FPS 2.5 ~2.5 0.95
LPIPS 0.0132 0.0186 0.0154
PSNR 33.69 32.67 33.10
SSIM 0.973 0.962 0.966
Loss 1 1 1
result (rgb)
nerfacto_rgb.mp4
nerfacto_tiny_rgb.mp4
kanerf_rgb.mp4
result (depth)
nerfacto_depth.mp4
nerfacto_tiny_depth.mp4
kanerf_depth.mp4

KAN has the potential for optimization, particularly with regard to accelerating its inference speed. We plan to develop a CUDA-accelerated version of KAN to further enhance its performance : D

  • The Visualization of KanNeRF
Alt text

Contact us

@Manual{,
   title = {Hands-On NeRF with KAN},
   author = {Delin Qu, Qizhi Chen},
   year = {2024},
   url = {https://github.com/Tavish9/KANeRF},
 }