-
Notifications
You must be signed in to change notification settings - Fork 188
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add BAAI/bge-small-en-v1.5 Optimization #1634
base: main
Are you sure you want to change the base?
Conversation
"calibrate_method": "MinMax", | ||
"quant_preprocess": true, | ||
"prepare_qnn_config": true, | ||
"op_types_to_quantize": [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have a dev branch where I introduce an option called op_type_to_exclude which is used to modify op_types_to_quantize and nodes_to_exclude.
Olive/olive/passes/onnx/quantization.py
Line 57 in 6e5c3b4
"op_types_to_exclude": PassConfigParam( |
Looks like it might be useful here too when it gets merged
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
otherwise, we need to know all of the op types present in the model.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
currently use append_first_op_types_to_quantize_list with nodes_to_exclude will do this. Will we also update this logic?
Olive/olive/passes/onnx/quantization.py
Line 409 in 043f7e1
if run_config["append_first_op_types_to_quantize_list"]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
honestly, I am not sure why this option was added and if it is used for anything right now.
Not sure if we will touch this option and related logic but I plan to update the logic to be able to use op_types_to_exclude
and nodes_to_exclude
. The op_types_to_exclude
has been very useful for me when I know I don't want to quantize all nodes for an op.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also created this PR in ort microsoft/onnxruntime#23779.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we could merge following the clip?
Describe your changes
Checklist before requesting a review
lintrunner -a
(Optional) Issue link