[QUESTION] Memory management suggestion

[yswhynot]

Dear amazing warp team,

I'm working on a design optimization project for tendon-driven soft fingers, adapted from the example_walker.py code. Although everything works fine for the forward and backward gradients, I noticed very heavy memory usage of the cuda graph. To get my soft structures to work reliably with the euler integrator, I had to set super large frame rate (1000) and substeps (300). Otherwise, the contacts between soft and rigid bodies get unreasonable. This makes it only feasible to run a short period of time without running into memory issue on RTX 4090.

I noticed the memory usage happens mostly on creating the cuda graph. May I know is there any suggestions or pointers over what's a better option?

My soft mesh is already the simplest possible, ~4k tetrahedron.

Thank you so much for your help!

[shi-eric]

Hi @yswhynot, are you sure that it's the use of CUDA graphs specifically that's causing the memory issue? It seems to me that it's more likely that the number of simulation states you need to keep in memory in order to run the backward pass is probably the real culprit. Have you tried running the problem with CUDA graph capture disabled?

If it's simply that you can't fit all the simulation states in memory, you would have to implement something like a gradient checkpointing strategy.

[yswhynot]

Hi Eric,

Oh yes I'm sorry, you are absolutely right. It is the list of states causing the memory issue.

If this is the case, do you have any recommendations of gradient checkpointing with warp? Or are there better ways to either turn something off to save the memory each states need, or better tuning to simulate soft-rigid contacts more stably with larger steps?

Thank you so much for your help!

[eric-heiden]

Hi @yswhynot,

We currently don't have an example for gradient checkpointing, we will consider adding this in the future.

Another thing worth trying is to improve the discretization of the mesh to make sure the tetrahedra are not degenerate (too small, almost planar, etc.). I noticed the voxelized discretization that we have in the bear model from example_walker.py seems to be particularly efficient to simulate for this reason.

[yswhynot]

Thank you so much for your suggestion!

My tetrahedra are indeed quite small because I am simulating the tendon's tension forces along the soft finger, so I need to put vertices all along the way and add forces to those vertices. Or if there is better ways of simulating tendons, I'd be happy to know!