Issue about the interpolated weights of patch embeding when swiching the patch size

[ShileiCao]

How should we scale the interpolated weights of patch embeding e.g., when decreasing patch size from 4 to 2?

[wesselb]

Hey @ShileiCao! If you change the patch size, you will need to fine-tune the model for a little bit to allow the model to adapt to the new patch size. Just changing the patch size and interpolating the weights of the patch embedding will not give performant predictions.