Trainer initializing multiple copies of model on single GPU

[naveenkumarmarri]

I'm trying to train a model using ddp on single machine multi GPU. I'm seeing that multiple copies of the model is being initialized on single GPU. This is leading to cuda out of memory issues.

Lightning invocation

    model = TrainDistributed()
    early_stop_callback = EarlyStopping(
        monitor='loss',
        min_delta=0.001,
        patience=3,
        verbose=False,
        mode='min'
    )
    checkpoint_callback = ModelCheckpoint(
        dirpath='models/',
        verbose=True,
        monitor='loss',
        save_top_k=3,
        mode='min'
    )
    
    tb_logger = pl_loggers.TensorBoardLogger('logs/')
    trainer = pl.Trainer(
        gpus=8,
        checkpoint_callback=checkpoint_callback,
        callbacks=[early_stop_callback],
        distributed_backend='ddp',
        # plugins=[DDPPlugin(find_unused_parameters=True)],
        logger=tb_logger,
        # precision=16
        )
    trainer.fit(model)