Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GPU] Improve sdpa_opt kernel performance with flashattn2 softmax tricks. #28013

Merged
merged 16 commits into from
Jan 7, 2025
Merged

[GPU] Improve sdpa_opt kernel performance with flashattn2 softmax tricks. #28013

merged 16 commits into from
Jan 7, 2025

Conversation

ceciliapeng2011
Copy link
Contributor

@ceciliapeng2011 ceciliapeng2011 commented Dec 11, 2024 •
edited by sshlyapn
Loading

Details:

  • Switch to FlashAttn2 softmax update tricks which reduces the number of non-matmul FLOPS.

Tickets:

  • 158462

@ceciliapeng2011 ceciliapeng2011 requested review from a team as code owners December 11, 2024 06:42
@github-actions github-actions bot added the category: GPU OpenVINO GPU plugin label Dec 11, 2024
@ceciliapeng2011 ceciliapeng2011 changed the title [GPU]optimize sdpa_opt kernel softmax. [GPU] Improve sdpa_opt kernel performance with flashattn2 softmax tricks. Dec 11, 2024
@sammysun0711
Copy link
Collaborator

Please rebase to latest master to includes this WA to build VS 2022: #28021

sshlyapn
sshlyapn approved these changes Dec 20, 2024
View reviewed changes
Copy link
Contributor

@sshlyapn sshlyapn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@sshlyapn sshlyapn added this to the 2025.0 milestone Dec 20, 2024
@ceciliapeng2011 ceciliapeng2011 mentioned this pull request Jan 3, 2025
Open
riverlijunjie
riverlijunjie approved these changes Jan 7, 2025
View reviewed changes
Copy link
Contributor

@riverlijunjie riverlijunjie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@p-durandin p-durandin added this pull request to the merge queue Jan 7, 2025
Merged via the queue into openvinotoolkit:master with commit 26e5fe9 Jan 7, 2025
167 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@p-durandin p-durandin p-durandin approved these changes

@riverlijunjie riverlijunjie riverlijunjie approved these changes

@sshlyapn sshlyapn Awaiting requested review from sshlyapn

Assignees

@sshlyapn sshlyapn

Labels
category: GPU OpenVINO GPU plugin
Projects
None yet
Milestone
2025.0
Development

Successfully merging this pull request may close these issues.
6 participants
@ceciliapeng2011 @sammysun0711 @sshlyapn @riverlijunjie @p-durandin @peterchen-intel