How pl synchronise objects and metrics when using DDP?

[heng-yuwen]

I want to calculate the confusion matrix and accumulate them at each step. And after each epoch, I calculate the accuracy and other segmentation metrics at the end of each epoch. However, it seems that with DDP mode, the metrics are synchronised at each step, so after calling .compute(), the final value is multiplied by the number of processors. Here is a sample code that I use to create the ConfusionMatrix object and how to update the value:

self.confmat_train = ConfusionMatrix(num_classes=self.n_class)
self.confmat_valid = ConfusionMatrix(num_classes=self.n_class)
self.confmat_test = ConfusionMatrix(num_classes=self.n_class)

# at validation_step
self.confmat_valid(pred, truth)

# at validation_epoch_end
self.confmat_valid.compute()

It seems that the object self.confmat_valid is automatically synchronised at each step, so when calling compute, pl sums self.confmat_valid.confmat across multiple GPUs, but all the confmats are the same. Therefore, the final result is multiplied by the number of processors. Is this behaviour desired or did I made any mistakes?

By the way, I use ddp_cpu mode to debug.

[SkafteNicki]

Metric will only sync after each step if you initialize them with dist_sync_on_step=True. Else they will sync whenever metric.compute is called. The fact that all your confusion matrices are the same may be coincidence or it may have something to do with ddp_cpu. I just re-ran some local code, and I can atleast say that there should be no problem using confusion matrix with standard ddp or ddp_spawn, where I observe the confusion matrices are different across processes and correctly reduced.

[heng-yuwen]

Then the problem may appear in ddp_cpu mode only. In ddp_cpu mode, all objects initialised as tensor are synchronised at each step. For example, define a = torch.tensor(0), with 2 processors. Then, at the training step, call a += 1. Assume train with 2 steps, the value of a should be 2. However, it is 4 in ddp_cpu mode.

[heng-yuwen]

I use Pytorch-lightning 1.2.1 and Pytorch 1.7.0

[heng-yuwen]

Or maybe they are not really "synchronised", maybe they are the same object in ddp_cpu mode. I guess this is the real situation.

[SkafteNicki]

Do you see a difference for the first epoch compared to all other epochs. In your example:
epoch 0 would return 4
epoch 1 would return 2
epoch 2 would return 2
...