Failed with error out of memory

[oramla]

Hello,
My problem is that I get this error when the testbed.exe has to exit.
I don't have any software spending resources and the memory is free.
The card I have is a Geforce Nvidia RTX 3060Ti, 32 Gb RAM and Procador Ryzen 7.
I have been changing the scale parameters in the transforms.js file from 12 to 2.

The error:
(ngp) E:\TUTORIAL\ngp\instant-ngp>E:\TUTORIAL\ngp\instant-ngp\build\testbed.exe --scene data/escalera
21:39:07 INFO Loading NeRF dataset from
21:39:07 INFO data\escalera\transforms.json
21:39:08 SUCCESS Loaded 86 images of size 4000x3000 after 0s
21:39:08 INFO cam_aabb=[min=[-0.840304,-2.04571,0.612019], max=[1.83417,3.2325,1.5317]]
21:39:09 ERROR Uncaught exception: Could not allocate memory: E:\TUTORIAL\ngp\instant-ngp\dependencies\tiny-cuda-nn\include\tiny-cuda-nn/gpu_memory.h:109 cudaMalloc(&rawptr, n_bytes+DEBUG_GUARD_SIZE*2) failed with error out of memory

Thank you very much for the awesome work.

[Tom94]

Hi there, instant-ngp stores training images internally using fp16 values per channel, even if they come from .jpg/.png, which means your 86 images occupy 86 * 4000 * 3000 * 4 * 2 = 8.2 GB, which is more RAM than your GPU has (8 GB).

You could try loading only half of those images and I suspect the rest of the algorithm will fit in the remaining space.

[Tom94]

Gladly!

I wouldn't recommend dabbling too much in the mesh export function, though. While it works in principle, there are many shortcomings (like the ones you mentioned) which can't be fixed by simple configuration changes. We merely provide marching cubes a the "obvious baseline" of extracting a mesh from a NeRF.

For anything more practically oriented, I'd be looking at a dedicated paper for mesh reconstruction, such as https://nvlabs.github.io/nvdiffrec/

[felixgeen]

Hi Tom,

When I test this formula I find it doesn't make sense for my dataset. I have a dataset of 690 images at 2560x2560 resolution.

690 * 2560 * 2560 * 4 * 2 = 36.1GB

By this calculation I would need a GPU with 36GB of VRAM to load this dataset but I am able to load it on my 4090 with only 24GB. Is there something I am missing about the formula?

Thanks for your help!

[Tom94]

Yes; I've improved the algorithm since I wrote the above comment, such that it keeps 8bit RGBA images in 8bit format internally. I.e. you don't need the multiplication by 2 if you're loading JPG or PNG, resulting in 18GB rather than 36GB.

[felixgeen]

Thanks Tom, and how much VRAM would you say is required to be left as headroom for Windows/instant-ngp algorithm etc?

[doctorjana]

Thank you very much. It is working on my GTX 1050 Ti now.

[StarsTesla]

So if there is a way to put images into model batch by batch?Or it's already in pybind?

[cduguet]

I already asked this somewhere else , but a way I imagine could be useful would be to downsample images after being positioned in COLMAP. This would need of course, to edit transforms.json as well. I wonder if someone has done this already, either in this repo or in other NeRF flavour.

[felixgeen]

this would be so useful! Is there any way of resizing a dataset after alignment?

[GabrielePaolini]

Hello,
Thank you in advance for your amazing work and support here.
I'm encountering the same error while testing both the fox dataset and the ficus dataset (100 images 800x800).
Despite having only a NVIDIA GeForce GTX 1050 Ti (4Gb RAM), I get the same CUDA_ERROR_OUT_OF_MEMORY even though I tried reducing the number of images loaded (to the point of having only 4 images), reducing their resolution, reducing the GUI resolution and reducing the aabb_scale parameter to 1 or 2, as suggested in other issues.

Assuming the formula given by @Tom94 is valid, the memory should be sufficient. I don't understand where's the source of problem.
I should also mention that the computer I run the tests with is an ASUS TUF FX505GE-BQ151T with a second integrated INTEL UHD Graphics 630 card.
Could it be the cause of this error?

Thank you in advance for you answers.

[JiayingGaoo]

great work, i meet the same problem with you. Is there some solution for this problem?

[doctorjana]

I have the same graphic card and the same issue. Hope there will be a fix soon.

[doctorjana]

The size is evident in OPs success message, but not in your(as well as my) success message.

This must be something.
I'd be grateful if there is some fix to this.

However I see a spike in the GPU usage whenever I run the testbed, so it could be a memory issue too. But then again I tried with the lowest possible resolution and 10 images, still no luck.

[HenReeco]

I am having a similar issue but I was actually running the same exact script all day with no issues. it certainly is large but the machine handled it pretty well. then i just ran the same exact thing again but this time saving the snapshot and i doubt that is what caused the issue.
Even other files like the colmap2nerf.py run into memory issues.

My guess is that somehow memory is not being released and I don't know how to do it. My machine is pretty powerful, it's a server with an A100 and 40GB memory. It's a multi-user server but nobody is running anything actively right now.
Any suggestions?