Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QNN EP] Fix multithread sync bug in ETW callback #23156

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

adrianlizarraga
Copy link
Contributor

@adrianlizarraga adrianlizarraga commented Dec 19, 2024

Description

Fixes crash in QNN dlls when an ETW callback tries to change the QNN log level. This is caused by a function that does not lock a mutex before modifying the QNN log level.

Motivation and Context

An ETW callback into QNN EP leads to a crash within QNN SDK dlls. It happens approximately 1 out of 3 full QNN unit tests runs.

The cause is a multithreading synchronization bug in QNN EP. We're not always locking a mutex when ETW calls QNN EP to notify of ETW config change.

There are two branches in the QNN EP callback function that try to update the QNN log handle. One branch correctly locks a mutex, but other does not lock it at all. This causes crashes within QNN dlls.

The fix is to lock the mutex in both paths.

(void)qnn_backend_manager_->UpdateQnnLogLevel(ortETWSeverity);
(void)qnn_backend_manager_->ResetQnnLogLevel(ortETWSeverity);
Copy link
Contributor Author

@adrianlizarraga adrianlizarraga Dec 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: This is the root of the problem. The UpdateQnnLogLevel function was not locking a mutex.

I combined UpdateQnnLogLevel() and ResetQnnLogLevel() into a single function that locks the mutex and handles both updating and resetting.

@adrianlizarraga adrianlizarraga marked this pull request as ready for review December 19, 2024 20:50
@adrianlizarraga
Copy link
Contributor Author

Hi @ivberg, would appreciate your review for this ETW bug fix for QNN EP if able.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants