Finetune video predictor on custom data

[ahmadsadeed]

I was able to finetune SAM on custom image segmentation dataset which trained the mask decoder and got very high IoU. Then I used the same model weights to test video predictor but didn't see improvement. I assume it has to do with memory encoder. How can I finetune the whole model? Didn't want to reinvent the wheel if someone has done it.