Skip to content

Commit

Permalink
Pass attribute to from_pretrained
Browse files Browse the repository at this point in the history
  • Loading branch information
NielsRogge committed Aug 29, 2024
1 parent 7d93bb5 commit abdf5c5
Showing 1 changed file with 3 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -150,13 +150,13 @@ def convert_blip2_checkpoint(
if "opt" in model_name:
tokenizer = AutoTokenizer.from_pretrained("facebook/opt-2.7b")
elif "itm" in model_name:
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased", truncation_side="right")
tokenizer = BertTokenizer.from_pretrained(
"bert-base-uncased", truncation_side="right", model_input_names=["input_ids", "attention_mask"]
)
tokenizer.add_special_tokens({"bos_token": "[DEC]"})
else:
tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-xl")

tokenizer.model_input_names = ["input_ids", "attention_mask"]

if "itm" in model_name:
eos_token_id = None
else:
Expand Down

0 comments on commit abdf5c5

Please sign in to comment.