Add Paligemma 1 support

[tboinovski1]

Overview

Adds support for Google's PaliGemma 1 vision-language model to both the HF→GGUF converter and the MTMD. This is a start but WIP as it will likely need discussion and some help from maintainers to make sure the implementation fits llama.cpp better.

Since SigLIP and Gemma1 are already supported by llama.cpp so my goal has been to reuse existing functionality as much as possible while adding support for Paligemma 1 too.

I can provide f16 ggufs but they can also be generated using the convert tools with the changes in the PR.

Model highlights:
Blog: https://huggingface.co/blog/paligemma
Paper: https://arxiv.org/abs/2407.07726

Working sample (image):

./llama.cpp/build/bin/llama-mtmd-cli   -m gguf/paligemma-3b-mix-224-f16.gguf   --mmproj gguf/paligemma-3b-mix-224-mmproj-f16.gguf   --image test-224x224-1.jpg   -p "what do you see in the image?"   --chat-template paligemma -n 32
0.00.155.147 I common_init_result: fitting params to device memory ...
0.00.155.151 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.00.924.432 I common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
0.00.976.144 I mtmd_cli_context: chat template example:
Hello
Hi there
How are you?

0.01.061.913 I main: loading model: gguf/paligemma-3b-mix-224-f16.gguf
0.01.061.918 W WARN: This is an experimental CLI for testing multimodal capability.
0.01.061.918 W       For normal use cases, please use the standard llama-cli

kitten and puppy

Requirements

• I have read and agree with the contributing guidelines: YES
• AI usage disclosure: Yes, to reason about convert & mtmd, testing and code review.

[max-krasnyansky]

@ngxson Looks like @tboinovski1 missed AI disclosure note. Sorry about that. Otherwise this is legit. AI assisted but otherwise nothing crazy.
Ok to reopen?

[ngxson]

@max-krasnyansky I cannot, this is badly hallucinated PR. The impl works but is incorrect. If the author really know what's they doing, they should (1) not use AI to write the description and (2) should refer to prior discussions about the model.

if you really care about mtmd, please don't push PR without fully understand it

[ngxson]

Again, letting AI to write PR description is a direct violation of project's policy. @max-krasnyansky you are a maintainer, please adhere to the policy and if you really have a good reason not to - please explain

[max-krasnyansky]

@max-krasnyansky I cannot, this is badly hallucinated PR. The impl works but is incorrect. If the author really know what's they doing, they should (1) not use AI to write the description and (2) should refer to prior discussions about the model.

if you really care about mtmd, please don't push PR without fully understand it

Oh. Interesting. Hmm. We'll follow up. Must be missing something but it's not obvious. Sorry.
The intent is to add proper Paligemma support. If you have any specific pointers please do share otherwise we'll comb through prior history.

[max-krasnyansky]

Again, letting AI to write PR description is a direct violation of project's policy. @max-krasnyansky you are a maintainer, please adhere to the policy and if you really have a good reason not to - please explain

Yes yes. Sorry, again no ill intent here. Maybe overly verbose. We'll work on updating that.

[ngxson]

As mentioned both in the policy and guidelines : search for prior PRs and issues before working on a feature. A quick search for issues containing the word "paligemma" should be the easiest thing to try

[tboinovski1]

@max-krasnyansky I cannot, this is badly hallucinated PR. The impl works but is incorrect. If the author really know what's they doing, they should (1) not use AI to write the description and (2) should refer to prior discussions about the model.

if you really care about mtmd, please don't push PR without fully understand it

@ngxson Absolutely no bad intent here. I'm not an expert on convert and MTMD but I'm learning and I'm just trying to help. I've worked on this for a while and done quite a bit of testing. It's not perfect but I'm willing to put in the effort to make it work if you have some pointers on what has to be changed.

Should I provide more working examples/samples or other info?

[tboinovski1]

As mentioned both in the policy and guidelines : search for prior PRs and issues before working on a feature. A quick search for issues containing the word "paligemma" should be the easiest thing to try

I searched PRs before I started working on this and only found PR #7553 which was from '24 and still unmerged (currently can't see the code changes). And recently #22528 but that was for Paligemma 2. Reading through both, what makes this implementation incorrect. I also saw a few old FRs for Paligemma support. What am I missing?

@ngxson Could you reopen this? If you give me pointers what needs fixing/rework, I will fix it. Paligemma 1 might be an old model by today's standards but there is still quite a bit of interest is seeing it supported in llama.cpp

[ngxson]

How do you "trying to help and learn" but pushing a slop PR and did not even read the code change from the PR that you mentioned? I don't see how it's not a bad intention, you directly violated multiple points in the contribution guidelines (not researching prior discussions, AI-generated descriptions, no proper AI usage disclosure)

If paligemma is tha easy to implement, it should already supported a long time ago. Unfortunately it requires some complicated addition APIs to make it work. So no, this is not a good first issue.

And do not reach out to maintainers via email unless absolutely necessary. You are not the only contributor here, if you want to help, don't push slop code and let other contributors who has real interest to work on it.

[tboinovski1]

How do you "trying to help and learn" but pushing a slop PR and did not even read the code change from the PR that you mentioned? I don't see how it's not a bad intention, you directly violated multiple points in the contribution guidelines (not researching prior discussions, AI-generated descriptions, no proper AI usage disclosure)

I was aware of 7553 but saw 22528 yesterday. I assumed that 7553 was outdated. The violations were truly unintentional. After I kicked the PR off, I stepped out for a meeting and by the time I got back (30m) to review the description the PR was already closed.

If paligemma is tha easy to implement, it should already supported a long time ago. Unfortunately it requires some complicated addition APIs to make it work. So no, this is not a good first issue.

This is actually a useful piece if info to have up front. The 'complicated addition APIs' part. I did not get that out of the PRs. My bad.