Support for multiple optimizers

[linshokaku]

Want to be able to apply different optimizers to each weight of a single model.

example:

class MyModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.l1 = nn.Linear(20, 15)
        self.l2 = nn.Linear(15, 10)

        with torch.no_grad():
            self.l1.weight.copy_(torch.ones((15, 20)))
            self.l1.bias.copy_(torch.ones((15,)))
            self.l2.weight.copy_(torch.ones((10, 15)))
            self.l2.bias.copy_(torch.ones((10,)))

    def forward(self, x):
        y = F.relu(self.l1(x))
        y = self.l2(y)
        return y

model = MyModel()
adam_optimizer = torch.optim.Adam(model.l1.parameters(), lr=0.1)
sgd_optimizer = torch.optim.SGD(model.l2.parameters(), lr=0.1)

trainer = ppe.engine.create_trainer(
    model_with_loss, optimizer???, epochs,
    device=device,
)

[linshokaku]

The current CodeBlockLogic can support multiple optimizers, but it has lost the ability to specify the target optimizer by model_name, which is incompatible with Logic.

pytorch-pfn-extras/pytorch_pfn_extras/handler/_logic.py

Line 457 in cbdd4a2

list(optimizers.values()),

In Logic, only one can be specified by model_name, so I think it needs to be modified.

pytorch-pfn-extras/pytorch_pfn_extras/handler/_logic.py

Line 307 in cbdd4a2

optimizers[self.model_name].zero_grad()