-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add NativeScan name as filter for Gluten #694
Conversation
Signed-off-by: Allen Xu <[email protected]>
@@ -293,9 +293,10 @@ abstract class AppBase( | |||
val readSchema = ReadParser.formatSchemaStr(meta.getOrElse("ReadSchema", "")) | |||
val scanNode = allNodes.filter(node => { | |||
// Get ReadSchema of each Node and sanitize it for comparison | |||
// The "NativeScan" name is from Velox engine |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is very easy to be changed. Better to think of a better way for this. I realize this can be as a follow-up.
Signed-off-by: Allen Xu <[email protected]>
The intent is to be able to parse Velox plans and NativeScan is the only thing we don't support? ie everything else parses out fine. We should be added a test so if you have an eventlog that has this we should figure out where to add that. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @wjxiz1992 for putting up the PR.
- the PR is missing unit tests
- It is preferred for each PR to have an opened issue that describes the bug/feature-request so that it gets reviewed and prioritized along the other features.
cc @mattahrens Do we want to support nativeScan
?
I'm missing the context for this. Is there a corresponding issue? If this is to resolve a parsing failure to not fail for qual tool execution, then this is fine. However note that we have not benchmarked against Velox so any estimations out of the qual tool would likely not be as accurate. |
Yes, this MR is just a quick fix for a customer issue and it needs more code to really handle the problem:
I will polish this afterwards. |
I can include that change in the existing PR #709 @wjxiz1992 Can you show a snippet from eventlog that shows the nodeDescription containing |
@amahussein I am afraid that may contain customer sensitive info, let me send you the snippet in private. |
#723 supersedes this PR. |
When parsing eventlog generated by Gluten engine, the Scan node name starts with "NativeScan" instead of "Scan". This results the empty output after filtering.
Add the "NativeScan" for filter so it can be parsed successfully.