Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TESTS] Use FP32 inference precision, FP16 KV cache precision for pipelines #1485

Merged
merged 1 commit into from
Jan 6, 2025

Conversation

ilya-lavrenov
Copy link
Contributor

@ilya-lavrenov ilya-lavrenov commented Jan 6, 2025

OpenVINO plugins enable different kind of optimizations by default like KV cache compression to int8, fp16 inference precision, while in GenAI tests we want to test pipelines and how they are compared against HF / optimum w/o extra optimizations:

def get_default_properties():
import openvino.properties.hint as hints
import openvino as ov
return {
hints.inference_precision : ov.Type.f32,
hints.kv_cache_precision : ov.Type.f16,
}

Hopefully, we can merge int8 KV cache by default for CB then #1206, because in tests we will still compare FP16 KV cache, while official Validation should be responsible for validation against reference via WWB metrics.

@github-actions github-actions bot added the category: Python API Python API for GenAI label Jan 6, 2025
@ilya-lavrenov ilya-lavrenov force-pushed the default-config branch 4 times, most recently from a03d7cf to f86a642 Compare January 6, 2025 18:23
@github-actions github-actions bot removed the category: whisper Whisper pipeline label Jan 6, 2025
@ilya-lavrenov ilya-lavrenov disabled auto-merge January 6, 2025 18:26
@github-actions github-actions bot added the category: samples GenAI samples label Jan 6, 2025
@ilya-lavrenov ilya-lavrenov added this pull request to the merge queue Jan 6, 2025
@ilya-lavrenov ilya-lavrenov removed this pull request from the merge queue due to a manual request Jan 6, 2025
@ilya-lavrenov ilya-lavrenov merged commit 48dfd16 into openvinotoolkit:master Jan 6, 2025
59 checks passed
@ilya-lavrenov ilya-lavrenov deleted the default-config branch January 6, 2025 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants