How to set up DDP correctly if processes are created externally & CUDA_VISIBLE_DEVICES is set

[yongsiang-fb]

Hi,

I am trying to get PyTorch lightning work within a certain cluster environment.

In particular, the DDP processes would be created externally, and additionally, CUDA_VISIBLE_DEVICES will be set by the cluster manager so that only 1 device would be visible, which is the device the process is supposed to use.

I found that in this situation, if I define a subclass of ClusterEnvironment where local_rank is set as the real local rank of the process, an exception would be thrown because PyTorch Lightning would attempt to access self.parallel_devices[self.local_rank] but there is only 1 device present in self.parallel_devices because of the CUDA_VISIBLE_DEVICES.

What would be the best approach to make it work? Should I implement my own strategy class to override the behavior of self.parallel_devices[self.local_rank] ?

Thanks a lot!

[rohitgr7]

CUDA_VISIBLE_DEVICES will be set by the cluster manager so that only 1 device would be visible, which is the device the process is supposed to use.

Do you mean each node has 1 GPU visible? if yes, in such a case, local_rank should always return 0 and global_rank should return the rank across all nodes.

[yongsiang-fb]

Do you mean each node has 1 GPU visible?

No, each node will have multiple processes, but each of these process can only see 1 visible device.

For example,

• Node 1
  • Process 1 (local_rank=0, global_rank=0): CUDA_VISIBLE_DEVICES= 0
  • Process 2 (local_rank=1, global_rank=1): CUDA_VISIBLE_DEVICES= 1
• Node 2
  • Process 1 (local_rank=0, global_rank=2): CUDA_VISIBLE_DEVICES= 1
  • Process 2 (local_rank=1, global_rank=3): CUDA_VISIBLE_DEVICES= 2

[rohitgr7]

Process 1 (local_rank=0, global_rank=2): CUDA_VISIBLE_DEVICES= 1

local_rank and CUDA_VISIBLE_DEVICES don't match here.
is it possible to ensure all the devices are reflected by CUDA_VISIBLE_DEVICES?

for eg, on node1

• node1: CUDA_VISIBLE_DEVICES=0,1

else you have to override the Strategy class.

[yongsiang-fb]

Yeah, they do not match. The idea is that the cluster manager would assign the GPU for each process via CUDA_VISIBLE_DEVICES instead of allowing each process to choose by themselves so that the cluster manager can monitor the resource being used.

else you have to override the Strategy class.

Right. I am thinking of overriding the root_device method so that it would no longer use local_rank for indexing. But not sure if there are other places in PyTorch lightning that would be affected by this mismatch between local_rank & CUDA_VISIBLE_DEVICES.

[doruchan]

Hi @yongsiang-fb, sorry to resurrect this thread, but have you found a solution for this issue? I'm in a very similar situation, in which my cluster is managed by exposing only 1 GPU per process, and local_rank and CUDA_VISIBLE_DEVICES are not matching.

cheers