Skip to content

Commit

Permalink
adding pdf and links
Browse files Browse the repository at this point in the history
  • Loading branch information
cvachha committed Jan 1, 2024
1 parent 98ce881 commit 189f9da
Show file tree
Hide file tree
Showing 5 changed files with 11 additions and 12 deletions.
Binary file not shown.
17 changes: 8 additions & 9 deletions dreamcrafter_progress.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,23 @@ show_tile: false
</ul>

## Overview
For my masters thesis, I am trying to create a system based on the proposal I made in 2022 [here](nerfenvironmentcreation.html).
We propose Dreamcrafter, a Virtual Reality 3D content generation and editing system Assisted by generative AI. Our system addresses this gap by proposing a system that harnesses the immersive experience and spatial interactions of VR, coupled with the advanced capabilities of generative AI, to enhance the process of 3D environment creation and editing.
For my masters thesis, I am trying to create a system based on the proposal I made in 2022 [here](nerfenvironmentcreation.html). In my VR/AR class in my first semester (Fall 2023) of my 5th year masters program (1 yr program), to implement two prototype systems which leverage NeRFs, 3DGS, and Stable Diffusion to create a VR interface for 3D photo-realistic content creation. We call our system Dreamcrafter, a Virtual Reality 3D content generation and editing system Assisted by generative AI. Our system attempts addresses the gap in HCI-VR research in generative AI tools by proposing a system that enhances the process of 3D environment creation and editing.

NeRF and diffusion models offer unparalleled realism and detail in rendering, however, their integration into user-friendly platforms for 3D environment creation is still in its infancy.
Dreamcrafter aims to bridge these gaps by creating a seamless and intuitive interface for users to design, modify, and generate complex and photorealistic 3D environments. Editing of radiance fields and generative 3D objects is currently limited to text prompts or limited 2D interfaces. Current research in NeRFs and diffusion models is primarily on enhancing image/reconstruction quality, and we aim to address the noticeable lack of exploration in the application of user interfaces designed for editing and controllability of these models and novel 3D representations.
The core of our approach is a VR-based system that allows users to interact with and manipulate 3D objects and environments in real-time. Dreamcrafter involves two subsystems systems which leverage novel 3D representations and stable diffusion. The stable diffusion powered system assigns semantically mapped spatial tags to 3D primitive objects to generate stable diffusion previews of scenes. Our second subsystem leverages NeRFs and 3D Gaussian Splatting for rendering and editing of 3D photo realistic scenes. Dreamcrafter is designed to be simple to use, lowering the barrier to entry for users without extensive experience in 3D modeling, while still providing realistic output results.
Integration of NeRF/3DGS and diffusion models into user-friendly platforms for 3D environment creation is still in its infancy. Editing of radiance fields and generative 3D objects is currently limited to text prompts or limited 2D interfaces. Current research in NeRFs and diffusion models is primarily on enhancing image/reconstruction quality, and we aim to address the noticeable lack of exploration in the application of user interfaces designed for editing and controllability of these models and novel 3D representations.

The core of our approach is a VR-based system that allows users to interact with and manipulate 3D objects and environments in real-time. Dreamcrafter involves two subsystems systems which leverage novel 3D representations and stable diffusion. The stable diffusion powered system assigns semantically mapped spatial tags to 3D primitive objects to generate stable diffusion previews of scenes. Our second subsystem leverages NeRFs and 3D Gaussian Splatting for rendering and editing of 3D photo realistic scenes. Dreamcrafter is designed to be simple to use, lowering the barrier to entry for users without extensive experience in 3D modeling, while still providing realistic output results.

Our current system is
Our current system is a proof-of-concept prototype which we developed within a month.

We developed a few prototype systems within a month
## Prototype Demos
Here are some demo videos of our prototype systems that demonstrate a basic version of some of the key features/interactions.

## Future plans
Here is a NeRF/GS capture of our poster demo in our class:
<iframe src="https://lumalabs.ai/embed/afc9f2d5-a1bc-4681-914a-5dd157938e33?mode=sparkles&background=%23ffffff&color=%23000000&showTitle=true&loadBg=true&logoPosition=bottom-left&infoPosition=bottom-right&cinematicVideo=undefined&showMenu=true" width="281" height="500" frameborder="0" title="luma embed" style="border: none;"></iframe>

## Future plans

Here is our in-class assignment short paper.
Here is our in-class assignment short paper:



Expand Down
2 changes: 1 addition & 1 deletion nerfenvironmentcreation.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
layout: post
title: NeRF Environment Creation System
title: NeRF Environment Creation System Proposal (2022)

show_tile: false
---
Expand Down
2 changes: 1 addition & 1 deletion nerfstudio_contributions.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ show_tile: false
<li><a href="research.html" class="button small">Go back and see more research projects</a></li>
</ul>

Since Jan 2023 I have made contributions to the Nerfstudio API system including some features and small improvements to other parts. I am also acknowledged in the Nerfstudio SIGGRAPH paper.
Since Jan 2023 I have made contributions to the Nerfstudio API system including some features and small improvements to other parts. I am also acknowledged in the Nerfstudio SIGGRAPH paper and my Blender add-on is mentioned during presentations of Nerfstudio.

## Nerfstudio Blender VFX Add-on

Expand Down
2 changes: 1 addition & 1 deletion research.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ menu-show: true
<ul class="actions">
<li><a href="https://docs.nerf.studio/extensions/blender_addon.html" class="button">Documentation</a></li>
<li><a href="https://www.youtube.com/watch?v=A7La8tWp_0I" class="button">Video</a></li>
<li><a href="https://drive.google.com/file/d/1y2xVk228dabXHfzNOPyNzSh8YRVYGmSc/view?usp=sharing" class="button">View Abstract</a></li>
<li><a href="https://cvachha.github.io/assets/pdfs/nerf_vfx_abstract.pdf" class="button">View Abstract</a></li>
</ul>
</div>
</div>
Expand Down

0 comments on commit 189f9da

Please sign in to comment.