Optim deformable detr #33600

yonigozlan · 2024-09-19T17:34:47Z

What does this PR do?

This PR is part of an ongoing effort to optimize the inference speed of Transformers' vision models. (along with #33412 for now).
For more info on the specific issues these optimizations target and how they help improve the inference speed of compiled models, you can check out this Notion page.

The following metrics are for model inference only! Pre/post-processing time are not measured here. Currently, Transformers image processors seem to be a big bottleneck for inference speed, so end-to-end inference will sadly be much slower than this.

A few things to note:

Other models which use deformable attention were modified due to "copied from", but the modifications should be non breaking. I haven't measured it but those models could also be slightly faster due to it :).
Grounding Dino in particular could really benefit from more changes than those that were simply copied, but this could be part of another PR.
Contrary to the optimization done on RT-DETR, here both eager and compiled models are improved :).
I have no idea what is going on with the compiled fp16 model without custom kernel. The trace shows full use of the GPU, and similar changes were made on RT-DETR with considerable gain in the same settings. If anyone has a theory I would be glad to hear it :).
Deformable DETR (unlike RT-DETR) uses the custom kernel for deformable attention by default. But the user needs to have ninja-build installed for it to work.

Fixes # (issue)

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

@qubvel @molbap @amyeroberts @NielsRogge

HuggingFaceDocBuilderDev · 2024-09-19T18:00:09Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

amyeroberts

Looks great!

Just two things before being merge ready:

We'll need to run the slow tests for these models
Removing the baseline file

amyeroberts · 2024-09-19T18:00:26Z

src/transformers/models/deformable_detr/modeling_deformable_detr_baseline.py

qubvel

Looks good to me! Just some nits and questions :)

Can you please provide a bit more details on what was changed and why that leads to a better performance? As far as I see the general idea of this PR is to avoid .item() or am I missing anything else?

qubvel · 2024-09-19T18:49:48Z

src/transformers/models/deformable_detr/modeling_deformable_detr.py

+        y_embed = pixel_mask.cumsum(1, dtype=torch.float16)
+        x_embed = pixel_mask.cumsum(2, dtype=torch.float16)


Just wondering, why should we use float16 instead of float32? shouldn't it be pixel_values.dtype?

Oops absolutely, thanks for catching that.

qubvel · 2024-09-19T18:53:04Z

src/transformers/models/deformable_detr/modeling_deformable_detr.py

@@ -704,7 +708,8 @@ def forward(

        batch_size, num_queries, _ = hidden_states.shape
        batch_size, sequence_length, _ = encoder_hidden_states.shape
-        if (spatial_shapes[:, 0] * spatial_shapes[:, 1]).sum() != sequence_length:
+        total_elements = sum(shape[0] * shape[1] for shape in spatial_shapes_list)


in case it holds here :)

Suggested change

total_elements = sum(shape[0] * shape[1] for shape in spatial_shapes_list)

total_elements = sum(height * width for height, width in spatial_shapes_list)

HuggingFaceDocBuilderDev · 2024-10-01T12:49:37Z

Hey! 🤗 Thanks for your contribution to the transformers library!

Before merging this pull request, slow tests CI should be triggered. To enable this:

Add the run-slow label to the PR
When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command [run-slow] followed by a comma separated list of all the models to be tested, i.e. [run_slow] model_to_test_1, model_to_test_2
- If the pull request affects a lot of models, put at most 10 models in the commit message
A transformers maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is here.

amyeroberts

Thanks for adding and iterating!

* optimize deformable detr * fix copies * remove deformable_detr_basline * fix hardcoded float16 and .float() * [run slow] deformable-detr,grounding-dino,mask2former,oneformer,rt-detr * [run slow] deformable_detr,grounding_dino,mask2former,oneformer,rt_detr

philkuz · 2024-10-22T22:33:25Z

Hi, I'm coming to this PR a month late, but I noticed there are some changes here that I think would be great to transfer over to other models that are down-stream of DeformableDetr, specifically Mask2Former.

I would love to contribute these changes, but to do this properly, I'd love if I could get more context on a few things that I couldn't discover from this PR, because the Notion doc is not public :)

How do you run the eager/compiled versions of these models? I tried a very naive pass at pulling a DeformableDetr model into torch.compile with no arguments, but that failed. Is there another path?
What is the underlying motivation here? Do you see users compile these models to run them faster in production environments?

For context, I'm trying to use the new-ish torch.export() non-strict path to export Mask2Former and I noticed that the changes in this PR include some of the same changes I would need to make to the Mask2Former modeling code. I would love to contribute these changes and have them align with the optimization objective you have in this PR!

yonigozlan · 2024-10-24T23:27:04Z

Hi @philkuz!
Great to see some interest in propagating these changes and making Transformers models more torch compile compatible :).
Here is a published version of the notion page, I haven't had the time to polish it and it's not quite up to date, but hopefully it can help answer some of your questions.

As for the motivation, in general I think ensuring Transformers models are fully optimized for compilation sets a nice standard. There has been more and more issues and feature requests related to torch compilation, so there seems to be growing interest in using it in production. With torch.compile and torch.export gaining traction, especially for performance-critical applications like real-time vision, I can see these optimizations becoming increasingly valuable.

yonigozlan added 2 commits September 19, 2024 16:12

optimize deformable detr

1224075

fix copies

205d1a0

amyeroberts reviewed Sep 19, 2024

View reviewed changes

src/transformers/models/deformable_detr/modeling_deformable_detr_baseline.py Outdated

Copy link

Collaborator

amyeroberts Sep 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To delete?

remove deformable_detr_basline

6902c7b

amyeroberts added the run-slow label Sep 19, 2024

qubvel approved these changes Sep 19, 2024

View reviewed changes

yonigozlan added 2 commits September 20, 2024 02:41

fix hardcoded float16 and .float()

b1c23db

[run slow] deformable-detr,grounding-dino,mask2former,oneformer,rt-detr

6144271

[run slow] deformable_detr,grounding_dino,mask2former,oneformer,rt_detr

0b16150

amyeroberts approved these changes Oct 2, 2024

View reviewed changes

yonigozlan merged commit ee71c98 into huggingface:main Oct 2, 2024
25 of 27 checks passed

philkuz mentioned this pull request Oct 24, 2024

fix(Mask2Former): torch export #34393

Open

5 tasks

yonigozlan mentioned this pull request Nov 1, 2024

Compile Grounding DINO #34556

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optim deformable detr #33600

Optim deformable detr #33600

yonigozlan commented Sep 19, 2024

HuggingFaceDocBuilderDev commented Sep 19, 2024

amyeroberts left a comment

amyeroberts Sep 19, 2024

qubvel left a comment

qubvel Sep 19, 2024

yonigozlan Sep 19, 2024

qubvel Sep 19, 2024 •

edited

Loading

HuggingFaceDocBuilderDev commented Oct 1, 2024

amyeroberts left a comment

philkuz commented Oct 22, 2024 •

edited

Loading

yonigozlan commented Oct 24, 2024

		y_embed = pixel_mask.cumsum(1, dtype=torch.float16)
		x_embed = pixel_mask.cumsum(2, dtype=torch.float16)

	total_elements = sum(shape[0] * shape[1] for shape in spatial_shapes_list)
	total_elements = sum(height * width for height, width in spatial_shapes_list)

Optim deformable detr #33600

Optim deformable detr #33600

Conversation

yonigozlan commented Sep 19, 2024

What does this PR do?

Before submitting

Who can review?

HuggingFaceDocBuilderDev commented Sep 19, 2024

amyeroberts left a comment

Choose a reason for hiding this comment

amyeroberts Sep 19, 2024

Choose a reason for hiding this comment

qubvel left a comment

Choose a reason for hiding this comment

qubvel Sep 19, 2024

Choose a reason for hiding this comment

yonigozlan Sep 19, 2024

Choose a reason for hiding this comment

qubvel Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Oct 1, 2024

amyeroberts left a comment

Choose a reason for hiding this comment

philkuz commented Oct 22, 2024 • edited Loading

yonigozlan commented Oct 24, 2024

qubvel Sep 19, 2024 •

edited

Loading

philkuz commented Oct 22, 2024 •

edited

Loading