enable low-precision pipeline #31625

jiqing-feng · 2024-06-26T03:57:56Z

As the previous PR #31444 mentioned, it could enable low-precision pipelines by converting the outputs to float(). I followed the codes in here. Do you mind taking a review? Thx!

aliencaocao · 2024-06-26T13:58:37Z

Related to #31342 but I dont quite get your changes - what exactly does it fix? When I tested all the pipelines in fp16 none of them had issues outputting logits

jiqing-feng · 2024-07-04T03:22:14Z

Related to #31342 but I dont quite get your changes - what exactly does it fix? When I tested all the pipelines in fp16 none of them had issues outputting logits

Hi @aliencaocao , FP16 works fine but BF16 is not acceptable for numpy, you can see:

SunMarc

Thanks for addding this @jiqing-feng ! Left a few comments ! Nice tests =)

src/transformers/pipelines/automatic_speech_recognition.py

src/transformers/pipelines/image_classification.py

src/transformers/pipelines/token_classification.py

src/transformers/pipelines/zero_shot_classification.py

jiqing-feng · 2024-07-05T03:07:10Z

Hi @SunMarc. I have fixed all your comments; please review them. BTW, the failed tests are due to the connection error, not related to my changes.

SunMarc

Thanks for iterating ! Just a few nits !

src/transformers/pipelines/image_classification.py

src/transformers/pipelines/token_classification.py

src/transformers/testing_utils.py

Co-authored-by: Marc Sun <[email protected]>

SunMarc

Thanks for iterating @jiqing-feng !

jiqing-feng · 2024-07-10T05:22:04Z

Hi @amyeroberts , would you please review this PR? Thx!

HuggingFaceDocBuilderDev · 2024-07-10T11:49:21Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

jiqing-feng · 2024-07-16T02:02:36Z

Hi @amyeroberts . I have passed all tests, do you mind take a review and merge it? Thx!

jiqing-feng · 2024-07-17T01:36:53Z

Hi @amyeroberts . This PR should be ready to merge, please take a review, thx!

amyeroberts · 2024-07-17T16:50:18Z

Hi @jiqing-feng, thanks for opening this PR! I'll get to reviewing it soon, but will likely be in a few days.

jiqing-feng · 2024-07-30T01:09:55Z

Hi @amyeroberts . Do you mind reviewing this PR? Thx.

jiqing-feng · 2024-08-06T02:07:04Z

Hi @amyeroberts . Do you think it could be merged?

yao-matrix · 2024-08-19T02:05:16Z

@amyeroberts , could you help review this PR? Thx.

yao-matrix · 2024-09-13T00:45:23Z

@SunMarc , do you have a suggestion any other people from HF can help review and merge this PR? Seems @amyeroberts has no bandwidth on this these serval months. Thx.

LysandreJik · 2024-09-17T10:07:09Z

Sorry for the delay @yao-matrix!

@Rocketknight1 is managing the pipeline; Matt, would you mind reviewing this PR when you have a second?

Rocketknight1

Overall this PR seems good to me! However, I prefer if outputs.dtype in (torch.bfloat16, torch.float16) rather than if outputs.dtype == torch.bfloat16, so we can catch float16 as well.

Other than that, I'm happy with it!

Rocketknight1 · 2024-09-17T13:44:15Z

src/transformers/pipelines/image_classification.py

+        if self.framework == "pt" and outputs.dtype == torch.bfloat16:
+            # To enable using bf16
+            outputs = outputs.float().numpy()


This seems like a regression to me! The previous code worked with both torch.bfloat16 and torch.float16.

I use torch 2.4, numpy-2.1.1 in python 3.10 and it failed:

jiqing-feng · 2024-09-18T01:45:12Z

Overall this PR seems good to me! However, I prefer if outputs.dtype in (torch.bfloat16, torch.float16) rather than if outputs.dtype == torch.bfloat16, so we can catch float16 as well.

Other than that, I'm happy with it!

Hi @Rocketknight1 , thanks for your review, I have fixed it by your comments.

jiqing-feng · 2024-09-18T06:54:59Z

Hi @Rocketknight1 , do you mind helping to re-run the tests? Thx.

Rocketknight1 · 2024-09-18T15:08:06Z

Hi @jiqing-feng, another PR at #33554 touched the same files. I'm sorry - I didn't realize that it was doing the same thing as this PR! That PR has been merged, so I've merged this PR with the code there to avoid conflicts.

Rocketknight1 · 2024-09-18T15:45:39Z

@jiqing-feng tests pass now and this looks good - are you okay for us to merge it?

Also cc @LysandreJik for final review, but no rush!

jiqing-feng · 2024-09-19T00:38:00Z

@jiqing-feng tests pass now and this looks good - are you okay for us to merge it?

Yes, please.

LysandreJik

That works for me! Thank you @jiqing-feng!

* enable low-precision pipeline * fix parameter for ASR * reformat * fix asr bug * fix bug for zero-shot * add dtype check * rm useless comments * add np.float16 check * Update src/transformers/pipelines/image_classification.py Co-authored-by: Marc Sun <[email protected]> * Update src/transformers/pipelines/token_classification.py Co-authored-by: Marc Sun <[email protected]> * fix comments * fix asr check * make fixup * No more need for is_torch_available() --------- Co-authored-by: Marc Sun <[email protected]> Co-authored-by: Matt <[email protected]> Co-authored-by: Matt <[email protected]>

jiqing-feng added 5 commits June 26, 2024 07:50

enable low-precision pipeline

4579ef1

fix parameter for ASR

c3ac276

reformat

9105ad4

fix asr bug

3fab613

fix bug for zero-shot

159bb2d

jiqing-feng closed this Jul 4, 2024

jiqing-feng reopened this Jul 4, 2024

LysandreJik requested a review from SunMarc July 4, 2024 12:44

SunMarc reviewed Jul 4, 2024

View reviewed changes

jiqing-feng added 3 commits July 4, 2024 22:23

add dtype check

9d12a01

rm useless comments

9bb1d45

add np.float16 check

a3a5be5

SunMarc reviewed Jul 5, 2024

View reviewed changes

src/transformers/pipelines/image_classification.py Outdated Show resolved Hide resolved

src/transformers/pipelines/token_classification.py Outdated Show resolved Hide resolved

src/transformers/testing_utils.py Show resolved Hide resolved

jiqing-feng and others added 2 commits July 9, 2024 08:58

Update src/transformers/pipelines/image_classification.py

73e8879

Co-authored-by: Marc Sun <[email protected]>

Update src/transformers/pipelines/token_classification.py

5480f41

Co-authored-by: Marc Sun <[email protected]>

SunMarc approved these changes Jul 9, 2024

View reviewed changes

SunMarc requested a review from amyeroberts July 9, 2024 11:30

Merge branch 'huggingface:main' into pipeline

a7d08e3

Merge branch 'huggingface:main' into pipeline

4b06b63

Rocketknight1 approved these changes Sep 17, 2024

View reviewed changes

Merge branch 'huggingface:main' into pipeline

46dbf62

jiqing-feng and others added 4 commits September 18, 2024 05:35

fix comments

76e91dc

fix asr check

94cc8b3

Merge branch 'main' into pipeline

daa467c

make fixup

0a2ba5b

No more need for is_torch_available()

98959f3

LysandreJik approved these changes Sep 20, 2024

View reviewed changes

LysandreJik merged commit 49a0bef into huggingface:main Sep 20, 2024
22 checks passed

jiqing-feng mentioned this pull request Oct 29, 2024

enable QA bf16 pipeline #34483

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enable low-precision pipeline #31625

enable low-precision pipeline #31625

jiqing-feng commented Jun 26, 2024

aliencaocao commented Jun 26, 2024 •

edited

Loading

jiqing-feng commented Jul 4, 2024 •

edited

Loading

SunMarc left a comment

jiqing-feng commented Jul 5, 2024

SunMarc left a comment

SunMarc left a comment

jiqing-feng commented Jul 10, 2024

HuggingFaceDocBuilderDev commented Jul 10, 2024

jiqing-feng commented Jul 16, 2024

jiqing-feng commented Jul 17, 2024

amyeroberts commented Jul 17, 2024

jiqing-feng commented Jul 30, 2024

jiqing-feng commented Aug 6, 2024

yao-matrix commented Aug 19, 2024

yao-matrix commented Sep 13, 2024

LysandreJik commented Sep 17, 2024

Rocketknight1 left a comment

Rocketknight1 Sep 17, 2024

jiqing-feng Sep 18, 2024 •

edited

Loading

jiqing-feng commented Sep 18, 2024 •

edited

Loading

jiqing-feng commented Sep 18, 2024 •

edited

Loading

Rocketknight1 commented Sep 18, 2024

Rocketknight1 commented Sep 18, 2024

jiqing-feng commented Sep 19, 2024

LysandreJik left a comment

enable low-precision pipeline #31625

enable low-precision pipeline #31625

Conversation

jiqing-feng commented Jun 26, 2024

aliencaocao commented Jun 26, 2024 • edited Loading

jiqing-feng commented Jul 4, 2024 • edited Loading

SunMarc left a comment

Choose a reason for hiding this comment

jiqing-feng commented Jul 5, 2024

SunMarc left a comment

Choose a reason for hiding this comment

SunMarc left a comment

Choose a reason for hiding this comment

jiqing-feng commented Jul 10, 2024

HuggingFaceDocBuilderDev commented Jul 10, 2024

jiqing-feng commented Jul 16, 2024

jiqing-feng commented Jul 17, 2024

amyeroberts commented Jul 17, 2024

jiqing-feng commented Jul 30, 2024

jiqing-feng commented Aug 6, 2024

yao-matrix commented Aug 19, 2024

yao-matrix commented Sep 13, 2024

LysandreJik commented Sep 17, 2024

Rocketknight1 left a comment

Choose a reason for hiding this comment

Rocketknight1 Sep 17, 2024

Choose a reason for hiding this comment

jiqing-feng Sep 18, 2024 • edited Loading

Choose a reason for hiding this comment

jiqing-feng commented Sep 18, 2024 • edited Loading

jiqing-feng commented Sep 18, 2024 • edited Loading

Rocketknight1 commented Sep 18, 2024

Rocketknight1 commented Sep 18, 2024

jiqing-feng commented Sep 19, 2024

LysandreJik left a comment

Choose a reason for hiding this comment

aliencaocao commented Jun 26, 2024 •

edited

Loading

jiqing-feng commented Jul 4, 2024 •

edited

Loading

jiqing-feng Sep 18, 2024 •

edited

Loading

jiqing-feng commented Sep 18, 2024 •

edited

Loading

jiqing-feng commented Sep 18, 2024 •

edited

Loading