how to set find_unused_parameters=True?

[xiadingZ]

🐛 Bug

RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your 
module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the 
keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` 
function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel 
module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss 
function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).

when I use pt 1.0.8, my model is ok, but when I switch to 1.1.4, it throws this error. It seems 1.0.8 enable unused parameters by default, but 1.1.4 not. How to solve this problem.

I think switch find_unused_parameters=True by default to False is a breaking change, but in docs, it doesn't mention, yet no clear instructions to set to True .

[siyuanfeng-tri]

i ran into the same issue. any ideas?

[xiadingZ]

    class MyDDP(DDPPlugin):

        def configure_ddp(self, model, device_ids):
            model = LightningDistributedDataParallel(model, device_ids, find_unused_parameters=True)
            return model

    my_ddp = MyDDP()

    trainer = Trainer.from_argparse_args(hparams,
                                         callbacks = [lr_logger, checkpoint_callback],
                                         plugins = [my_ddp])

this is my workaround. but hasn't tested whether it will influence my model precision.

[johngrabner]

xiadingZ, does your solution go into PyTorch lightning or as an application?

[xiadingZ]

In changelog, it doesn't show have fixed this bug

[carmocca]

No need to subclass DDPPlugin. This is enough:

trainer = pl.Trainer(plugins=[DDPPlugin(find_unused_parameters=True)])

Which is used here:

https://github.com/PyTorchLightning/pytorch-lightning/blob/0a50bb406fa41dfa6a0e2be52f531a9c81c87d00/pytorch_lightning/plugins/ddp_plugin.py#L66-L68

Sorry for the inconvenience!

[DuinoDu]

When using

trainer = pl.Trainer(plugins=[DDPPlugin(find_unused_parameters=True)])

extra args parallel_devices is also required for DDPPlugin. For example:

parallel_device_ids = device_parser.parse_gpu_ids(lightning_config['gpus'])
parallel_devices = [torch.device("cuda", i) for i in parallel_device_ids]
lightning_config['plugins'] = [DDPPlugin(parallel_devices, find_unused_parameters=True)]

BTW, if other default args to DDPPlugin is changed, such as cluster_environment and sync_batchnorm, also need to set here, which is not very convenient.

[carmocca]

Should be solved by #6051

[rubencart]

Please, when you make changes like this, could you just update the documentation immediately? Like, in the same PR? This is the I don't know how many-th time I have to go search in github issues and discussions how to fix my code to work with PL updates that are already in the code but not in the documentation... Imo, this would improve your usability a looot

[kwea123]

@carmocca What do you mean "should be solved"? Do you mean adding trainer = pl.Trainer(plugins=[DDPPlugin(find_unused_parameters=True)]) is enough? I still encounter this error if I didn't set anything.

[carmocca]

I have to go search in github issues and discussions how to fix my code to work with PL updates that are already in the code but not in the documentation

Sorry for the inconvenience! If you feel like we are missing a particular piece of information in the docs, we'd appreciate you opening a PR adding it where you think is appropriate.

@carmocca What do you mean "should be solved"? Do you mean adding trainer = pl.Trainer(plugins=[DDPPlugin(find_unused_parameters=True)]) is enough? I still encounter this error if I didn't set anything.

I was referring to having to set parallel_devices by "should be solved"

find_unused_parameters was changed as it is the recommended default by PyTorch as it can incur a large performance hit