Outline of current research project on creating a NeRF creation system for VR
+Outline propsal of current research project on creating a NeRF creation system for VR created in 2022
+
+ Since Jan 2023, I have been contributing features to the Nerfstudio system including the Blender VFX add-on and VR180/Omnidirectional (VR 360) video/image render outputs.
+
@@ -28,7 +44,7 @@ show_tile: false
A few experiements including virtual real time backgrounds and virtual metahuman actors
Renders of select Neural Radiance Fields I captured from Luma Labs AI, NVidia Instant NeRF, and NerfStudio
+Renders of select Neural Radiance Fields I captured from Luma Labs AI, NVidia Instant NeRF, and NerfStudio
+
+ Will soon be updated with a selection from (hundreds of) my 2023 captures
Blog style post explaining the timeline of my interest and motivation in NeRFs, lightfields, and neural rendering
+Blog style post explaining the timeline of my interest and motivation in NeRFs, lightfields, and neural rendering I wrote in 2022
diff --git a/nerfstudio_contributions.md b/nerfstudio_contributions.md index 03a90eb..c65c8a5 100644 --- a/nerfstudio_contributions.md +++ b/nerfstudio_contributions.md @@ -15,13 +15,13 @@ Since Jan 2023 I have made contributions to the Nerfstudio API system including -I created a Blender add-on that allows NeRFs to be used in visual effects. This enables a pipeline for integrating NeRFs into traditional compositing VFX pipelines using Nerfstudio. This approach leverages using Blender, a widely used open-source 3D creation software, to align camera paths and composite NeRF renders with meshes and other NeRFs, allowing for seamless integration of NeRFs into traditional VFX pipelines. It allows for more controlled camera trajectories of photorealistic scenes, compositing meshes and other environmental effects with NeRFs, and compositing multiple NeRFs in a single scene. This approach of generating NeRF aligned camera paths can be adapted to other 3D tool sets and workflows, enabling a more seamless integration of NeRFs into visual effects and film production. This also supports Nerfstudio gaussian splatting as well. +I created a Blender add-on that allows NeRFs to be used in visual effects. This enables a pipeline for integrating NeRFs into traditional compositing VFX pipelines using Nerfstudio. This approach leverages Blender, a widely used open-source 3D creation software, to align camera paths and composite NeRF renders with meshes and other NeRFs, allowing for seamless integration of NeRFs into traditional VFX pipelines. It allows for more controlled camera trajectories of photorealistic scenes, compositing meshes and other environmental effects with NeRFs, and compositing multiple NeRFs in a single scene. This approach of generating NeRF aligned camera paths can be adapted to other 3D tool sets and workflows, enabling a more seamless integration of NeRFs into visual effects and film production. This also supports Nerfstudio gaussian splatting as well. The exported mesh or point cloud representation is imported into Blender and a render camera path is generated by transforming the coordinate space of the NeRF scene to the Blender virtual camera, allowing aligned camera paths.-I created documentation for it [here](https://docs.nerf.studio/extensions/blender_addon.html) and a tutorial video demonstrating basic exmaples using the add-on as well as a breakdown of other effects that can be done with it. +I created documentation for it [here](https://docs.nerf.studio/extensions/blender_addon.html) and a tutorial video demonstrating basic examples using the add-on as well as a breakdown of other effects that can be done with it. diff --git a/research.md b/research.md index 85783cf..371ef07 100644 --- a/research.md +++ b/research.md @@ -35,7 +35,7 @@ menu-show: truev0.1.16 Released 🎉
— nerfstudio (@nerfstudioteam) January 27, 2023
New Blender integration for VFX workflows!#NeRF #nerfacto #VFX #Blender3d pic.twitter.com/uU8QO1AWwU
We propose a method for editing 3D Gaussian Splatting (3DGS) scenes with text-instructions in a method similar to Instruct-NeRF2NeRF. Given a 3DGS scene of a scene and the collection of images used to reconstruct it, our method uses an image-conditioned diffusion model (InstructPix2Pix) to iteratively edit the input images while optimizing the underlying scene, resulting in an optimized 3D scene that respects the edit instruction. We demonstrate that our proposed method is able to edit large-scale, real-world scenes, and is able to accomplish more realistic, targeted edits than prior work.
- - Paper comming soon
+ - Paper coming soon
- Nerfstudio integration supported
In my current 5th-year masters program (1 year graduate program after 4 year undergrad degree), I am attempting to build the initial concept of the VR environment creation system I proposed in 2022. For my VR/AR class I worked with a team to implement two prototype systems which leverage NeRFs, 3DGS, and Stable Diffusion to create a VR interface for 3D photo-realistic content creation. This includes a system to edit existing NeRF/GS scenes through voice, hand controls, and existing diffusion models (such as Instruct-Pix2Pix). We also have a system leveraging ControlNet to create 2D mockups of scenes based on 3D primitive objects. I am currently devloping the complete system with intelligent natural langue region selection and additional features. We are working towards a research publication for 2024.
+In my current 5th-year masters program (1 year graduate program after 4 year undergrad degree), I am attempting to build the initial concept of the VR environment creation system I proposed in 2022. For my VR/AR class I worked with a team to implement two prototype systems which leverage NeRFs, 3DGS, and Stable Diffusion to create a VR interface for 3D photo-realistic content creation. This includes a system to edit existing NeRF/GS scenes through voice, hand controls, and existing diffusion models (such as Instruct-Pix2Pix). We also have a system leveraging ControlNet to create 2D mockups of scenes based on 3D primitive objects. I am currently developing the complete system with intelligent natural language region selection and additional features. We are working towards a research publication for 2024.
+ -->
+