Adding new zero-shot examples

[SangbumChoi]

What does this PR do?

Fixes #32459

Following two PR should be merged before merging this example!
#31828
#31964

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[SangbumChoi]

@qubvel Hi Pavel this PR is ready for review. However, there is one problem with using evaluation process in Trainer.

In order to use the Trainer from the transformers we have to get input_ids for the post_processing pipeline which is from the tokenizer output. However, transformers TrainingArgument include_input_for_metric is set to get input variable of first index of class variable which is pixel_values. This is hard-coded so I cannot get the input_ids for the input. Also cc @muellerzr.

transformers/src/transformers/trainer.py

Line 3865 in aefd3e2

main_input_name = getattr(self.model, "main_input_name", "input_ids")

There is three ways for this

1. having additional argument to set the name
2. change the order of forward() (not preferred)
3. change to main_input_name = getattr(self.model, "input_ids", "input_ids")
   WDYT?

[qubvel]

Hi @SangbumChoi, thanks for working on this!

Is there any standard batch size that hugginface suggest? (In my experience maximum batch size was 2 with 24GB) The reason why I am asking is unlike standard detr, GroundingDino cannot handle batch size 8 for normal single GPU.

I think it is fine to have batch size 2 if the model can be trained with such a batch size. In README and a comment we can mention how much memory one would require to run the script and also advise using gradient accumulation, lower precision, and multi-GPU setup (if it is supported).

[qubvel]

However, there is one problem with using evaluation process in Trainer.
In order to use the Trainer from the transformers we have to get input_ids for the post_processing pipeline which is from the tokenizer output. However, transformers TrainingArgument include_input_for_metric is set to get input variable of first index of class variable which is pixel_values. This is hard-coded so I cannot get the input_ids for the input.

We can probably move these lines

main_input_name = getattr(self.model, "main_input_name", "input_ids")
inputs_decode = self._prepare_input(inputs[main_input_name]) if args.include_inputs_for_metrics else None

to a separate method of Trainer, for example, _get_inputs(...). Then we will be able to override it in a new ZeroShotTrainer class in a script to return required inputs.

[muellerzr]

agreed re; get_inputs

[SangbumChoi]

Let me work on pavel's suggestion and ping again. Thanks for the direction!

[SangbumChoi]

@qubvel Hi requesting for first review. I will state changes from the original object_detection.py

1. Instead of wandb logging graph, I just added table of GPU consumption (since there is no other architecture to compare)
2. Also as you might know calculating mAP from zero-shot is not accurate. Currently I have made workaround which is the logic that if the output text is directly in label2id then match otherwise treat as background object as 0. (We can discuss this if it seems inappropriate). See def convert_zero_shot_to_coco_format
3. Added ZeroShotTrainer to override Trainer with def _get_input_by_name

Also kudos to @EduardoPach for backbone_freeze and text_backbone_freeze. Which make the output more stable :)

Before finetune

After finetune

[qubvel]

Hi @SangbumChoi, thanks for iterating! Great work!

I left a comment regarding Trainer, also don't know if it deserves a separate PR. Overall looks very good to me, but I didn't do an in-depth view yet (let's wait for loss PR to be merged). For these examples the tests have to be added and information in README (see files in this directory). Also, feel free to add resources to docs of Grounding DINO. Btw, did you try to fine-tune OwlV2?

[SangbumChoi]

@qubvel Sounds good. I can wait until the loss PR get merged.

Btw, did you try to fine-tune OwlV2?

I didn't aware that we can fine-tune OwlV2, let me see while waiting loss PR!

[EduardoPach]

@SangbumChoi Not sure if this will still work if random_text_promt=True because the class_labels from targets will not necessarily follow label2id