Skip to content

compare-commits.sh: support both llama-bench and test-backend-ops #14392

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

yeahdongcn
Copy link
Collaborator

Make sure to read the contributing guidelines before submitting a PR

This is a follow-up to #14368, adding support for comparing test-backend-ops performance results between two commits.

Testing Done

Generated Tables

cd /Users/yexiaodong/go/src/github.com/ggerganov/llama.cpp && python3 scripts/compare-llama-bench.py -b 1d5f25c53 -c ecd7fdb4c --tool test-backend-ops -i ./test-backend-ops.sqlite
| Backend   | Operation   | Parameters                              |   Bandwidth (GB/s) 1d5f25c53 |   Bandwidth (GB/s) xd/compare |   Speedup |
|:----------|:------------|:----------------------------------------|-----------------------------:|------------------------------:|----------:|
| Metal     | ADD         | type=f32,ne=[4096,1,1,1],nr=[1,1,1,1]   |                        28.42 |                         28.45 |      1.00 |
| Metal     | ADD         | type=f32,ne=[4096,1,1,1],nr=[1,512,1,1] |                        86.60 |                         97.14 |      1.12 |

cd /Users/yexiaodong/go/src/github.com/ggerganov/llama.cpp && python3 scripts/compare-llama-bench.py -b ecd7fdb4c -c ecd7fdb4c --tool test-backend-ops -i ./test-backend-ops.sqlite                 
| Backend   | Operation   | Parameters                                                                       |   GFLOPS xd/test-backend-ops_sql |   GFLOPS xd/test-backend-ops_sql |   Speedup |
|:----------|:------------|:---------------------------------------------------------------------------------|---------------------------------:|---------------------------------:|----------:|
| Metal     | MUL_MAT     | type_a=f16,type_b=f32,m=128,n=1,k=16416,bs=[8,1],nr=[4,1],per=[0,1,2,3],v=1      |                           127.90 |                           127.90 |      1.00 |
| Metal     | MUL_MAT     | type_a=f16,type_b=f32,m=16416,n=1,k=128,bs=[8,1],nr=[4,1],per=[0,2,1,3],v=0      |                            33.98 |                            33.98 |      1.00 |
| Metal     | MUL_MAT     | type_a=f16,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0     |                            57.91 |                            57.91 |      1.00 |
| Metal     | MUL_MAT     | type_a=f16,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0     |                           115.00 |                           115.00 |      1.00 |

Generated Plot

plot

@yeahdongcn yeahdongcn requested a review from Copilot June 26, 2025 11:28
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for comparing performance results from both llama-bench and test-backend-ops by introducing tool-specific database schemas, CLI argument parsing, and formatting functions. Key changes include:

  • Refactoring database field and key property definitions to support both tools.
  • Updating table queries and input file handling based on a new --tool argument.
  • Enhancing the CLI script (compare-commits.sh) to allow selection of the tool and additional arguments.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
scripts/compare-llama-bench.py Adjusts SQLite table creation, queries, and result formatting for dual tool support.
scripts/compare-commits.sh Updates argument parsing and build/run logic to handle multiple tools.

@github-actions github-actions bot added script Script related python python script changes labels Jun 26, 2025
@yeahdongcn yeahdongcn marked this pull request as ready for review June 26, 2025 11:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
python python script changes script Script related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant