Manually syncing gradients in DDP with manual_backward #6340
Unanswered
paramhanji
asked this question in
DDP / multi-GPU / multi-node
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I have a specific application that requires multiple gradient updates. Here is a basic example:
As you can see, I use
manual_backward()
to compute and add gradients from multiple passes and perform gradient descent at the end. Is there any way I can use multi-GPU training for such a training step? Surely I will run into race conditions if I attempt multi-GPU training without any modifications to the above snippet! Generally, computation and syncing of gradients across GPUs are handled automatically by lightning.The DDP docs state that:
I'd like to find out if pytorch lighting has any mechanism to do gradient syncing manually. Happy to hear about alternative approaches as well.
Beta Was this translation helpful? Give feedback.
All reactions