Deterministic DataLoader on DDP reads same data on all subprocesses

[mrmotallebi]

I've used seed_everything(7) to initially set the seed then passed deterministic=True, accelerator='ddp' to Trainer to have it run on 4 GPUs.
Then I load my map-style dataset using a plain DataLoader with shuffle=True, num_workers=10 .
Now what happens is that each of the forked DDP processes spin up N (here 10) worker processes to read the data. So total 4 x 10 DataLoader processes. I have tried setting up a worker_init_fn to see the seeds they each receive, and indeed the worker processes for each GPU get different seeds, but they are the same across worker processes of different GPUs. This causes each data item to be read 4 times (the number of GPU / DDP processes) which I checked in the dataset's __getitem__. So the indexes for example would look like [3,3,3,3,7,7,7,7,2,2,2,2,...].

What is the way to fix this? Shouldn't the DistributedSampler for DDP automatically get a seed based on the subprocess that it is forked on? (Similar to DataLoader's worker_init_fn)

[mrmotallebi]

Update: This seemed to be the issue if only settings seeds and setting Pytorch and Cuda for deterministic execution, but adding deterministirc=True to PL's Trainer object seems to have resolved the issue.