DDP number of training iterations independent of #GPUs?

[timothybrooks]

Hello--

I am using DDP and anywhere from 1-8 GPUs. The PL documentation explains that it will wrap my training DataLoader in a DistributedSampler which gives each process a unique partition of the dataset. In this case, since it is also my understanding that the batch size is per-GPU, I would expect the total number of iterations to decrease proportionally to the number of GPUs. However the number of training batch iterations is constant (in my case ~100k) regardless of how many GPUs I use. It is equal to dataset_size / batch_size while I expected it to be dataset_size / (batch_size * num_gpus).

Am I missing something here? Why isn't PL partitioning the dataset such that each process sees a unique subset?

[mrmotallebi]

May it be related to my post, that each GPU is loading the whole dataset, thus the number of iterations for your case is constant?

#6350

Edit: Saw that you had already reacted to it. :)

[timothybrooks]

Yes I think this is the same underlying issue!

[mrmotallebi]

I tried changing the number of GPUs for my own case and the number of iterations changed accordingly.

[mrmotallebi]

How are you setting the number of GPUs? You have to pass it to the Trainer() otherwise it won't have effect.

[timothybrooks]

I created the following issue:
#6423