Adding new vocab doesn't saved the model

[andymvp2018]

System Info

8 gpu on A100

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

on the finetuning.py script, and did
https://github.com/meta-llama/llama-recipes/blob/main/src/llama_recipes/finetuning.py#L188
tokenizer.add(['wreqw', 'ewqr', 'weqrqewrqw',...])
model.resize_token_embeddings(len(tokenizer))

But then after saving the model when it finish training, I convert it from FSDP into huggingface checkpoint, , and see that the
model.get_input_embeddings().weight.shape[0] is still pre-added tokenizer dimension, which means that the newly added model embeddings isn't being saved.

Error logs

N/A

Expected behavior

The model should have a larger embeddings dimension

[jeffxtang]

@wukaixingxp @mreso can you please take a look?

[wukaixingxp]

@andymvp2018 Can you show me the complete log of how you train the model, how you convert the FSDP to HF model? What command did you use?

[andymvp2018]

For the training, I just use that https://github.com/meta-llama/llama-recipes/blob/main/src/llama_recipes/finetuning.py#L188.

For converting FSDP to HF:

• https://github.com/meta-llama/llama-recipes/blob/main/src/llama_recipes/inference/checkpoint_converter_fsdp_hf.py

python src/llama_recipes/inference/checkpoint_converter_fsdp_hf.py --fsdp_checkpoint_path fsdp_path --consolidated_model_path hf_path

[andymvp2018]

I think the problem is probability due to https://github.com/meta-llama/llama-recipes/blob/main/src/llama_recipes/tools/convert_hf_weights_to_llama.py#L45,

here, we should also change the dimensionality of the model (i.e, adding new tokens and then resize)