Unable to train fintuned LoRA on forget

[shaswati1]

Getting below error while trying to train the finetuned (LoRA llama2) on forget set:
ValueError: Target module Dropout(p=0.05, inplace=False) is not supported. Currently, only the following modules are supported: torch.nn.Linear, torch.nn.Embedding, torch.nn.Conv2d, transformers.pytorch_utils.Conv1D.

can you please help me with this?

[gyuilLim]

Did you solve it?? @shaswati1

[gyuilLim]

I thought it was a Python version issue, so I upgraded to 3.9, but the same error still occurs.

So I guess that if a Dropout layer is not inside a Conv2D or Linear sequential layer and instead exists independently outside the sequential layer, dropout layer is not converted to lora_dropout

As a temporary workaround, I added the following code at line 225 in peft/tuners/lora/model.py:

else:
    if isinstance(target, nn.Dropout):
        return

Although this is not a fundamental solution, it resolves the issue for now.

[gyuilLim]

Please refer to this issue.

huggingface/peft#2286