Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimized inference of Cohere model on HPU #1329

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Optimized inference of Cohere model on HPU #1329

wants to merge 5 commits into from

Conversation

XinyuYe-Intel
Copy link
Contributor

What does this PR do?

Optimized inference of Cohere model on HPU.

Before submitting

  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

@jiminha
Copy link
Collaborator

jiminha commented Sep 17, 2024

Can you provide the perf diff and accuracy value before/after this patch?

@XinyuYe-Intel
Copy link
Contributor Author

Can you provide the perf diff and accuracy value before/after this patch?

Below is the perf before the optimization:
image

and after the optimization:
image

The command is:

python run_generation.py --model_name_or_path CohereForAI/c4ai-command-r-v01 --batch_size 1 --use_hpu_graphs  --bf16  --do_sample --use_kv_cache --token ...

@libinta
Copy link
Collaborator

libinta commented Sep 24, 2024

@XinyuYe-Intel can you rebase?

@XinyuYe-Intel
Copy link
Contributor Author

@XinyuYe-Intel can you rebase?

rebased.

ssarkar2
ssarkar2 reviewed Oct 11, 2024
View reviewed changes
Copy link
Collaborator

@ssarkar2 ssarkar2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@XinyuYe-Intel could you please resolve the conflicts on this PR, looks good otherwise

@XinyuYe-Intel
Copy link
Contributor Author

@XinyuYe-Intel could you please resolve the conflicts on this PR, looks good otherwise

Resolved conflicts.

@libinta libinta added run-test Run CI for PRs from external contributors and removed review wip labels Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ssarkar2 ssarkar2 ssarkar2 approved these changes

@bhargaveede bhargaveede Awaiting requested review from bhargaveede bhargaveede is a code owner

@vivekgoe vivekgoe Awaiting requested review from vivekgoe vivekgoe is a code owner

@regisss regisss Awaiting requested review from regisss regisss is a code owner

Assignees
No one assigned
Labels
run-test Run CI for PRs from external contributors
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@XinyuYe-Intel @jiminha @libinta @ssarkar2