使用Randeng-Pegasus-523M-Summary-Chinese生成摘要报错PegasusTokenizer' object has no attribute 'vocab

[flystarts]

我的代码来自 https://huggingface.co/IDEA-CCNL/Randeng-Pegasus-523M-Summary-Chinese

from transformers import PegasusForConditionalGeneration
from tokenizers_pegasus import PegasusTokenizer

model = PegasusForConditionalGeneration.from_pretrained("IDEA-CCNL/Randeng-Pegasus-523M-Summary-Chinese")
tokenizer = PegasusTokenizer.from_pretrained("IDEA-CCNL/Randeng-Pegasus-523M-Summary-Chinese")

text = "据微信公众号“界面”报道"
inputs = tokenizer(text, max_length=1024, return_tensors="pt")

summary_ids = model.generate(inputs["input_ids"])
print(tokenizer.batch_decode(summary_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0])

执行时报错：
tokenizer = PegasusTokenizer.from_pretrained("IDEA-CCNL/Randeng-Pegasus-523M-Summary-Chinese")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\Downloads\Fengshenbang-LM-main\Fengshenbang-LM-main.venv\Lib\site-packages\transformers\tokenization_utils_base.py", line 2110, in from_pretrained
return cls._from_pretrained(
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\Downloads\Fengshenbang-LM-main\Fengshenbang-LM-main.venv\Lib\site-packages\transformers\tokenization_utils_base.py", line 2336, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\Downloads\Fengshenbang-LM-main\Fengshenbang-LM-main\fengshen\examples\pegasus\tokenizers_pegasus.py", line 156, in init
super().init(
File "C:\Users\Administrator\Downloads\Fengshenbang-LM-main\Fengshenbang-LM-main.venv\Lib\site-packages\transformers\tokenization_utils.py", line 367, in init
self._add_tokens(
File "C:\Users\Administrator\Downloads\Fengshenbang-LM-main\Fengshenbang-LM-main.venv\Lib\site-packages\transformers\tokenization_utils.py", line 467, in _add_tokens
current_vocab = self.get_vocab().copy()
^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\Downloads\Fengshenbang-LM-main\Fengshenbang-LM-main\fengshen\examples\pegasus\tokenizers_pegasus.py", line 209, in get_vocab
return dict(self.vocab, **self.added_tokens_encoder)
^^^^^^^^^^
AttributeError: 'PegasusTokenizer' object has no attribute 'vocab'

[mephosto]

参考：#426