support MiniCPM-V-2.6

[tc-mb]

Dear llama.cpp Official,

Hi, I'm writing to address our new PR submission for integrating our model MiniCPM-V 2.6 into llama.cpp. MiniCPM-V 2.6 is the latest and most capable model in the MiniCPM-V series. This model is stronger and supports multi-images understanding and video understanding.

This version of the model supports video understanding, and I have implemented functions such as video frame extraction in my fork version. However, because ffmpeg is introduced, there may be many environment and compilation issues in other devices. Therefore, I think it can be divided into multiple PR submissions.

1. This PR will first submit the modification of the model, and I hope it can be merged soon, so that the community can use MiniCPM-V 2.6 by GGUF first.
2. And in the later PR, support for video formats will be submitted, and we can spend more time discussing how llama.cpp can better integrate the function implementation of video understanding.

Best regards,
MiniCPM-V Official ^_^

[yorkane]

waiting for merge

[HaishengLiang]

waiting for merge

[nanowell]

waiting for merge

[saket424]

I have opened an issue 9066 where I experienced a crash after this pull request was merged. The crash was unrelated to this miniCPM-V-2.6 model. I hope you can reproduce the error

[tc-mb]

I have opened an issue 9066 where I experienced a crash after this pull request was merged. The crash was unrelated to this miniCPM-V-2.6 model. I hope you can reproduce the error

Hello, I saw that the issue you mentioned was that llava would crash, but my update only involves the part of minicpmv. Although I am not sure about the issue problem, I feel that it may not be the problem with this branch.
Can you test whether this branch will also crash before being merged? Of course, if it is indeed a problem introduced by this PR, I will be very happy to help modify it.

[saket424]

@tc-mb
The crash is not directly related to your miniCPM2.6 PR other than there is no crash before your PR and a crash after your PR owing to some uninitialized variables

Here is a PR that appears to fix the issue I reported
#9082

Sorry for the false alarm

[tc-mb]

@tc-mb The crash is not directly related to your miniCPM2.6 PR other than there is no crash before your PR and a crash after your PR owing to some uninitialized variables

Here is a PR that appears to fix the issue I reported #9082

Sorry for the false alarm

I'm glad your problem was solved.

[x4080]

@tc-mb Can we use mini cpm with context cache ? So that we upload image once and ask for multiple question referring to the same image ?

[tc-mb]

@tc-mb Can we use mini cpm with context cache ? So that we upload image once and ask for multiple question referring to the same image ?

Yes, it's now storing cache.

You can run in interactive mode to ask multiple rounds of questions.

./llama-minicpmv-cli -m ../MiniCPM-V-2_6/model/ggml-model-Q4_K_M.gguf --mmproj ../MiniCPM-V-2_6/mmproj-model-f16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image xx.jpg -i

or modify the minicpmv-cli function (which is more like an example) to achieve the functionality you want.

[yizhangliu]

Eagerly awaiting...

[fairydreaming]

Is this line necessary? It overrides minicpmv_version value set in the command line when converting MiniCPM-V2.5 which results in a broken mmproj-model-f16.gguf.

[x4080]

@tc-mb Can we use mini cpm with context cache ? So that we upload image once and ask for multiple question referring to the same image ?

Yes, it's now storing cache.

You can run in interactive mode to ask multiple rounds of questions.

./llama-minicpmv-cli -m ../MiniCPM-V-2_6/model/ggml-model-Q4_K_M.gguf --mmproj ../MiniCPM-V-2_6/mmproj-model-f16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image xx.jpg -i

or modify the minicpmv-cli function (which is more like an example) to achieve the functionality you want.

cool, thats a great feature, thanks @tc-mb

[dewarrn1]

Very cool! Are GPU operations supported at this time?

[tc-mb]

Very cool! Are GPU operations supported at this time?

I have tested in Ubuntu + Nvidia(4090), it is available and speed looks good. You can use it in the following way.

make LLAMA_CUDA=1
And add appropriate ngl parameters, such as.
./llama-minicpmv-cli -m ../MiniCPM-V-2_6/model/ggml-model-Q4_K_M.gguf --mmproj ../MiniCPM-V-2_6/mmproj-model-f16.gguf -c 4096 --temp 0.7 --top-p 0.8 --top-k 100 --repeat-penalty 1.05 --image xx.jpg -p "What is in the image?" -ngl 100

[dewarrn1]

Awesome, thanks!

[saket424]

@tc-mb
Can you give us the usage for how to serve up minicpm2.6 using llama-server so we can send it openai compatible chat completion requests with base64 encoded images

[tc-mb]

@tc-mb Can you give us the usage for how to serve up minicpm2.6 using llama-server so we can send it openai compatible chat completion requests with base64 encoded images

Sorry, I didn't test the server method when I updated it, I will support this capability in the near future.