fix "Cannot copy out of meta tensor; no data!" issue for BartForConditionalGeneration model

[yao-matrix]

fix issue: #36247

cause of this issue:

bart-large-cnn checkpoint only has model.decoder.embed_tokens.weight while load_state_dict, but in BartModel implementation, both model.decoder.embed_tokens.weight and model.encoder.embed_tokens.weight tie to the shared.weights which doesn't have corresponding real weights in checkpoints, so when tie_weights are called, things are back to a void shared.weights which in meta device, then lead to the model.to issue

@SunMarc @Rocketknight1 , I don't know whether this PR meet your standard since it changed modeling code, pls let me know your thoughts, thx.

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the Ready for review button (at the bottom of the PR page).

[Rocketknight1]

hi @yao-matrix, this causes several other BART tests to fail! You can see the failures in the CI at the bottom of this post.

Some of the failures are just code style - these are simple and can be fixed with make style or make fixup after pip install transformers[quality]. However, the failures in tests_torch indicate that this PR breaks BART weight loading, probably because the old shared tensor doesn't match up with the architecture after your PR. We won't be able to merge the PR unless those errors are resolved, and I'm not sure that will be possible while self.shared is removed.

[SunMarc]

I don't think we want to change the modeling to fix one specific checkpoint even though I know that this is one of the most used one. Maybe other checkpoints have the shared module instead. Instead, maybe we can try to update _tie_weights instead by tying self.shared to self.decoder.embed_tokens instead if self.decoder.embed_tokens is not on meta device ?

[yao-matrix]

@Rocketknight1 @SunMarc , Marc's method is better, I used it to fix the issue, pls help review and comment, thx.

[SunMarc]

Thanks for fixing this ! Really appreciate it !

[yao-matrix]

@ArthurZucker , pls take a look, thx.