-
Notifications
You must be signed in to change notification settings - Fork 39
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Refactor Exec Parsers - unify the code calling similar methods
This PR is to remove redundant code in many ExecParsers. We have several ExecParsers with the same functionality in parse code. In this PR, following changes are done: 1. If the Execs don't have expressions, then execName is assigned and ExecInfo object is created from GenericExecParser. 2. If the Execs have expressions then reflection is used to assign the appropriate parser to be used. 3. If there is any additional calculation in the parse function, then only that part is overriden in the ExecParser and the rest of the function uses the GenereicExecParser base code 4.Removed all the files which are not needed anymore Signed-off-by: Niranjan Artal <[email protected]>
- Loading branch information
Showing
29 changed files
with
378 additions
and
1,106 deletions.
There are no files selected for viewing
117 changes: 117 additions & 0 deletions
117
core/src/main/resources/execParserMappings/execParser.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,117 @@ | ||
{ | ||
"parsers": [ | ||
{ | ||
"execName": "AggregateInPandas", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "ArrowEvalPython", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "CartesianProduct", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "Coalesce", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "CollectLimit", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "CustomShuffleReader", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "FlatMapGroupsInPandas", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "InMemoryTableScan", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "LocalLimit", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "GlobalLimit", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "MapInArrow", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "MapInPandas", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "PythonMapInArrow", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "Range", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "Sample", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "Union", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "WindowInPandas", | ||
"parseExpressions": false | ||
}, | ||
{ | ||
"execName": "Expand", | ||
"parseExpressions": true, | ||
"expressionParserMethod": "parseExpandExpressions" | ||
}, | ||
{ | ||
"execName": "Filter", | ||
"parseExpressions": true, | ||
"expressionParserMethod": "parseFilterExpressions" | ||
}, | ||
{ | ||
"execName": "Generate", | ||
"parseExpressions": true, | ||
"expressionParserMethod": "parseGenerateExpressions" | ||
}, | ||
{ | ||
"execName": "ObjectHashAggregate", | ||
"parseExpressions": true, | ||
"expressionParserMethod": "parseAggregateExpressions" | ||
}, | ||
{ | ||
"execName": "Project", | ||
"parseExpressions": true, | ||
"expressionParserMethod": "parseProjectExpressions" | ||
}, | ||
{ | ||
"execName": "SortAggregate", | ||
"parseExpressions": true, | ||
"expressionParserMethod": "parseAggregateExpressions" | ||
}, | ||
{ | ||
"execName": "Sort", | ||
"parseExpressions": true, | ||
"expressionParserMethod": "parseSortExpressions" | ||
}, | ||
{ | ||
"execName": "TakeOrderedAndProject", | ||
"parseExpressions": true, | ||
"expressionParserMethod": "parseTakeOrderedExpressions" | ||
}, | ||
{ | ||
"execName": "Window", | ||
"parseExpressions": true, | ||
"expressionParserMethod": "parseWindowExpressions" | ||
} | ||
] | ||
} |
41 changes: 0 additions & 41 deletions
41
.../src/main/scala/com/nvidia/spark/rapids/tool/planparser/AggregateInPandasExecParser.scala
This file was deleted.
Oops, something went wrong.
41 changes: 0 additions & 41 deletions
41
core/src/main/scala/com/nvidia/spark/rapids/tool/planparser/ArrowEvalPythonExecParser.scala
This file was deleted.
Oops, something went wrong.
41 changes: 0 additions & 41 deletions
41
core/src/main/scala/com/nvidia/spark/rapids/tool/planparser/CartesianProductExecParser.scala
This file was deleted.
Oops, something went wrong.
42 changes: 0 additions & 42 deletions
42
core/src/main/scala/com/nvidia/spark/rapids/tool/planparser/CoalesceExecParser.scala
This file was deleted.
Oops, something went wrong.
41 changes: 0 additions & 41 deletions
41
core/src/main/scala/com/nvidia/spark/rapids/tool/planparser/CollectLimitExecParser.scala
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.