Adding new zero-shot examples #32483
Conversation
Co-authored-by: amyeroberts <[email protected]>
Co-authored-by: amyeroberts <[email protected]>
Co-authored-by: amyeroberts <[email protected]>
Co-authored-by: amyeroberts <[email protected]>
Co-authored-by: amyeroberts <[email protected]>
Co-authored-by: amyeroberts <[email protected]>
Co-authored-by: amyeroberts <[email protected]>
Co-authored-by: amyeroberts <[email protected]>
Co-authored-by: amyeroberts <[email protected]>
Co-authored-by: amyeroberts <[email protected]>
…ach/transformers into grounding_examples
Co-authored-by: Sangbum Daniel Choi <[email protected]>
…ach/transformers into grounding_examples
…to grounding_examples
|
Hi @qubvel It is now ready to review this new example. For the CI error it says |
|
|
Thanks for share the finetune code. I test with the code https://github.com/SangbumChoi/transformers on branch grounding_examples.
Looks like I missed something... |
|
@junxnone Thanks for the interest. This is because due to difference between zero-shot to coco converted AP. I have fixed in my code but not updated in here :( |
| def convert_zero_shot_to_coco_format(predictions, label2id): | ||
| """ | ||
| Convert zershot format output to typical object detection format in order to calculate mAP. | ||
|
|
||
| Args: | ||
| predictions (Dict): Output of zero-shot object detection | ||
| e.g. {'scores': tensor([0.4786, 0.4379, 0.4760], device='cuda:0'), 'labels': ['a cat', 'a cat', 'a remote control'], 'boxes': tensor([[344.6973, 23.1085, 637.1817, 374.2748],[ 12.2690, 51.9104, 316.8564, 472.4341],[ 38.5870, 70.0092, 176.7755, 118.1748]], device='cuda:0')} | ||
| label2id (Dict): Dictionary of label to id mapping | ||
|
|
||
| Returns: | ||
| Dict: Output of zero-shot object detection | ||
| e.g. {'scores': tensor([0.4786, 0.4379, 0.4760], device='cuda:0'), 'labels': [1, 1, 2], 'boxes': tensor([[344.6973, 23.1085, 637.1817, 374.2748],[ 12.2690, 51.9104, 316.8564, 472.4341],[ 38.5870, 70.0092, 176.7755, 118.1748]], device='cuda:0')} | ||
|
|
||
| """ | ||
| # convert center to corners format | ||
| torch_label = [] | ||
| for prediction in predictions: | ||
| scores = prediction["scores"] | ||
| device = scores.device | ||
| labels = prediction["labels"] | ||
| for label in labels: | ||
| if label in label2id: | ||
| torch_label.append(label2id[label]) | ||
| else: | ||
| # Give background class | ||
| torch_label.append(0) | ||
| prediction["labels"] = torch.Tensor(torch_label).to(dtype=torch.int32).to(device) | ||
|
|
||
| return predictions |
There was a problem hiding this comment.
@SangbumChoi if this is the zero-shot to coco converted, your repo is same with here.
There was a problem hiding this comment.
I also checkout the pr, and got the same result.
There was a problem hiding this comment.
I was meaning I have my own code which is not commited here
| validation_transform_batch = partial( | ||
| augment_and_transform_batch, | ||
| transform=validation_transform, | ||
| processor=processor, | ||
| id2label=id2label, | ||
| label2id=label2id, | ||
| random_text_prompt=True, | ||
| ) |
There was a problem hiding this comment.
I have some confusion here. Why with random_text_prompt=True do I get bad results?
There was a problem hiding this comment.
This will be deprecated. You understand correctly
There was a problem hiding this comment.
@SangbumChoi thanks a ton for your work so far! I am looking to use this code to finetune GroundingDINO for a critical medical imaging project - and from my understanding of the thread here I am noting some issues I should be careful about. Do you have any points on 'things to remember' when using this code? Or if you have a latest clean version would be super useful to us if you can commit those updates. One other option I am considering is finetuning GroundingDINO using this repo; but would still prefer finetuning the model off transformers. Thanks!!
|
@qubvel @EduardoPach thanks for your work on this so far (and in general for the wider HF ecosystem :) ). Ended up here after digging around for a train script entrypoint to use for finetuning grounding dino tiny. Any updates on why this PR is not merged to the main branch? @EduardoPach could you comment on what training script you used to finetune grounding dino tiny to get this model? Thanks in advance!!! |
What does this PR do?
Fixes #32459
Following two PR should be merged before merging this example!
#31828
#31964
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.