Zero-shot finetuning examples

[SangbumChoi]

Feature request

@amyeroberts @qubvel @NielsRogge

Does HF team intersted in adding zero-shot finetuning example like https://github.com/SangbumChoi/transformers/tree/grounding_examples/examples/pytorch/zero-shot?

If the team merge these two following open PR then
#31828
#31964

we can finetune the groundingdino and write something like this.
https://blog.roboflow.com/how-to-fine-tune-paligemma/

I will left some model that you can run
Model : https://huggingface.co/danelcsb/grounding-dino-tiny-finetuned-cppe-5-10k-steps-no-trainer/tree/main

Motivation

People can use zero-shot model to finetune there own projects.

Your contribution

I have made the branch already and ready to generate PR. Need some more modification.
Before

After

[qubvel]

Hi @SangbumChoi, the finetuned model example looks great! We are definitely interested in such an example for fine-tuning and blogpost! Let me know if you need any assistance!

[zappy586]

Hi @SangbumChoi. First of all, It's great that now we can fine-tune zero-shot models like GDINO directly from transformers. But in the example picture above, It seems that the class names in the finetuned example are very incoherent in comparison to the first image. Why do you that that is? Do you think overfitting the model on the fine-tuning dataset fix this issue?

[SangbumChoi]

@zappy586 Thanks for the interest. That is actually nice catch. I also have realized that problem and fixed in the above PR (It was not overfitting, it was about text_encoder part). Please see the updated examples in the PR!