Cannot load InternVideo2-8b models' LLM weight

[EigenTom]

I was trying to run the demo for the following three model cards:

https://huggingface.co/OpenGVLab/InternVideo2_Chat_8B_InternLM2_5
https://huggingface.co/OpenGVLab/InternVideo2_chat_8B_HD
https://huggingface.co/OpenGVLab/InternVideo2-Chat-8B

And I found the following problems:

1. LLM weights cannot be loaded and inference result is empty.

I let huggingface transformers deal with model downloading and loading. After the model is being downloaded, when it is being loaded I received the following error for all three model variants:

trainable params: 42,532,864 || all params: 7,290,556,416 || trainable%: 0.5833966788413643
Loading checkpoint shards: 100%|██████████| 4/4 [00:09<00:00,  2.29s/it]
Some weights of the model checkpoint at /home/luyi/.cache/huggingface/hub/models--OpenGVLab--InternVideo2_chat_8B_HD/snapshots/2e5c769e707bfcfa20f05e8aeacbc1cc7b4fda4d were not used when initializing InternVideo2_VideoChat2: ['lm.base_model.model.lm_head.weight', 'lm.base_model.model.model.layers.0.mlp.down_proj.weight', 'lm.base_model.model.model.layers.0.mlp.gate_proj.weight', 'lm.base_model.model.model.layers.0.mlp.up_proj.weight', 'lm.base_model.model.model.layers.0.self_attn.k_proj.weight', 'lm.base_model.model.model.layers.0.self_attn.o_proj.weight', 'lm.base_model.model.model.layers.0.self_attn.q_proj.weight', 'lm.base_model.model.model.layers.0.self_attn.v_proj.weight', 'lm.base_model.model.model.layers.1.mlp.down_proj.weight', 'lm.base_model.model.model.layers.1.mlp.gate_proj.weight', 'lm.base_model.model.model.layers.1.mlp.up_proj.weight', 'lm.base_model.model.model.layers.1.self_attn.k_proj.weight', 'lm.base_model.model.model.layers.1.self_attn.o_proj.weight', 'lm.base_model.model.model.layers.1.self_attn.q_proj.weight', 'lm.base_model.model.model.layers.1.self_attn.v_proj.weight', 'lm.base_model.model.model.layers.10.mlp.down_proj.weight', 'lm.base_model.model.model.layers.10.mlp.gate_proj.weight', 'lm.base_model.model.model.layers.10.mlp.up_proj.weight', 'lm.base_model.model.model.layers.10.self_attn.k_proj.weight', 'lm.base_model.model.model.layers.10.self_attn.o_proj.weight', 'lm.base_model.model.model.layers.10.self_attn.q_proj.weight', 'lm.base_model.model.model.layers.10.self_attn.v_proj.weight', 'lm.base_model.model.model.layers.11.mlp.down_proj.weight', 'lm.base_model.model.model.layers.11.mlp.gate_proj.weight', 'lm.base_model.model.model.layers.11.mlp.up_proj.weight', 'lm.base_model.model.model.layers.11.self_attn.k_proj.weight', 'lm.base_model.model.model.layers.11.self_attn.o_proj.weight', 'lm.base_model.model.model.layers.11.self_attn.q_proj.weight', 'lm.base_model.model.model.layers.11.self_attn.v_proj.weight', 'lm.base_model.model.model.layers.12.mlp.down_proj.weight', 
.........
'lm.base_model.model.model.layers.9.mlp.down_proj.weight', 'lm.base_model.model.model.layers.9.mlp.gate_proj.weight', 'lm.base_model.model.model.layers.9.mlp.up_proj.weight', 'lm.base_model.model.model.layers.9.self_attn.k_proj.weight', 'lm.base_model.model.model.layers.9.self_attn.o_proj.weight', 'lm.base_model.model.model.layers.9.self_attn.q_proj.weight', 'lm.base_model.model.model.layers.9.self_attn.v_proj.weight']
- This IS expected if you are initializing InternVideo2_VideoChat2 from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing InternVideo2_VideoChat2 from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of InternVideo2_VideoChat2 were not initialized from the model checkpoint at /home/luyi/.cache/huggingface/hub/models--OpenGVLab--InternVideo2_chat_8B_HD/snapshots/2e5c769e707bfcfa20f05e8aeacbc1cc7b4fda4d and are newly initialized: ['lm.base_model.model.lm_head.base_layer.weight', 'lm.base_model.model.model.layers.0.mlp.down_proj.base_layer.weight', 'lm.base_model.model.model.layers.0.mlp.gate_proj.base_layer.weight', 'lm.base_model.model.model.layers.0.mlp.up_proj.base_layer.weight', 'lm.base_model.model.model.layers.0.self_attn.k_proj.base_layer.weight', 'lm.base_model.model.model.layers.0.self_attn.o_proj.base_layer.weight', 'lm.base_model.model.model.layers.0.self_attn.q_proj.base_layer.weight', 'lm.base_model.model.model.layers.0.self_attn.v_proj.base_layer.weight', 'lm.base_model.model.model.layers.1.mlp.down_proj.base_layer.weight', 'lm.base_model.model.model.layers.1.mlp.gate_proj.base_layer.weight', 'lm.base_model.model.model.layers.1.mlp.up_proj.base_layer.weight', 'lm.base_model.model.model.layers.1.self_attn.k_proj.base_layer.weight', 'lm.base_model.model.model.layers.1.self_attn.o_proj.base_layer.weight', 'lm.base_model.model.model.layers.1.self_attn.q_proj.base_layer.weight', 'lm.base_model.model.model.layers.1.self_attn.v_proj.base_layer.weight', 
.........
'lm.base_model.model.model.layers.9.mlp.down_proj.base_layer.weight', 'lm.base_model.model.model.layers.9.mlp.gate_proj.base_layer.weight', 'lm.base_model.model.model.layers.9.mlp.up_proj.base_layer.weight', 'lm.base_model.model.model.layers.9.self_attn.k_proj.base_layer.weight', 'lm.base_model.model.model.layers.9.self_attn.o_proj.base_layer.weight', 'lm.base_model.model.model.layers.9.self_attn.q_proj.base_layer.weight', 'lm.base_model.model.model.layers.9.self_attn.v_proj.base_layer.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

which indicate the model weight of the entire LLM is not being loaded.
I have checked my account and confirm I have the access to Mistral-7B, InternLM 2.5, etc.

The model output result is shown as:

Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.

Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.

as well as

i.e. its output is empty. Such behaviour is consistent among all 3 models.

2. There's a bug in InternVideo2-Chat-8B's demo code:

as shown above, the tokenizer shall be instantiated after AutoTokenizer is being imported.

My environment is:
transformers == 4.40.0
peft == 0.14.0

I have also referred to this issue:
#154

while their problem is different than mine and no concrete solution is provided in the discussion.

Hope this issue can be sorted out, thank you for your support and patience.