Loading model with transformers / converting weights to GGUF

[vladislavdonchev]

We're trying to add support for Qwen2.5 VL in llama.cpp.

Taking into account the new architecture, we are able to extract and convert the VLM, but it seems that there is some incompatibility with the previous version.

This is the current SAFETENSORS -> GGUF code:
https://github.com/Independent-AI-Labs/llama.cpp/blob/master/examples/llava/qwen2_5_vl_surgery.py

The extracted model just crashes llama.cpp, so I am investigating that, but it is entirely possible that the model conversion is not handled correctly as well.

Any hints or pointers would be greatly appreciated! Thanks!

[vladislavdonchev]

UDPATE: OK, so I've noticed a difference in the patch merger compared to Qwen2:

  (merger): PatchMerger(
    (ln_q): LayerNorm((1280,), eps=1e-06, elementwise_affine=True)
    (mlp): Sequential(
      (0): Linear(in_features=5120, out_features=5120, bias=True)
      (1): GELU(approximate='none')
      (2): Linear(in_features=5120, out_features=3584, bias=True)
    )
  )

Affine transform for the normalization layer should not be a problem, but the out features are now 2048 instead of 3584.
We probably have to account for that somewhere along the line.

Are my assumptions correct? Could someone please advise?