-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HIVE-28259: Common table expression detection and rewrites using CBO #5249
Open
zabetak
wants to merge
10
commits into
apache:master
Choose a base branch
from
zabetak:cte-clean
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 5 commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
a3c72f3
HIVE-28259: Common table expression detection and rewrites using CBO
zabetak 5c5002b
HIVE-28259: Mark/Distinguish CTEs using Table#materializedTable field
zabetak 863551f
HIVE-28259: Print CTE suggestions only in cbo_query*q tests to avoid …
zabetak fc36fbd
HIVE-28259: Drop redundant null check in CommonRelSubExprRegisterRule
zabetak c71353e
HIVE-28259: Rename RemoveUnusedCteRule to RemoveInfrequentCteRule
zabetak 847fe8f
HIVE-28259: Consider CTE alias if available in table scan
zabetak a924080
HIVE-28259: Use Integer (instead of Long) to keep table occurrences
zabetak 88123b7
HIVE-28259: Ignore CTEs with non-INNER joins
zabetak c62adf3
HIVE-28259: Add test for CTEs and union branch pruning
zabetak 8e5869f
HIVE-28259: Missing new line EOF
zabetak File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
62 changes: 62 additions & 0 deletions
62
...a/org/apache/hadoop/hive/ql/optimizer/calcite/CommonTableExpressionIdentitySuggester.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one or more | ||
* contributor license agreements. See the NOTICE file distributed with | ||
* this work for additional information regarding copyright ownership. | ||
* The ASF licenses this file to you under the Apache License, Version 2.0 | ||
* (the "License"); you may not use this file except in compliance with | ||
* the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
package org.apache.hadoop.hive.ql.optimizer.calcite; | ||
|
||
import org.apache.calcite.plan.Contexts; | ||
import org.apache.calcite.plan.hep.HepPlanner; | ||
import org.apache.calcite.plan.hep.HepProgram; | ||
import org.apache.calcite.plan.hep.HepProgramBuilder; | ||
import org.apache.calcite.rel.RelNode; | ||
import org.apache.calcite.rel.metadata.RelMetadataQuery; | ||
import org.apache.hadoop.conf.Configuration; | ||
import org.apache.hadoop.hive.common.classification.InterfaceAudience; | ||
import org.apache.hadoop.hive.common.classification.InterfaceStability; | ||
import org.apache.hadoop.hive.ql.optimizer.calcite.rules.CommonRelSubExprRegisterRule; | ||
|
||
import java.util.Comparator; | ||
import java.util.List; | ||
import java.util.stream.Collectors; | ||
|
||
/** | ||
* Suggester for common table expressions that appear as is (identical trees) more than once in the query plan. | ||
*/ | ||
@InterfaceAudience.Public | ||
@InterfaceStability.Unstable | ||
public class CommonTableExpressionIdentitySuggester implements CommonTableExpressionSuggester { | ||
|
||
@Override | ||
public List<RelNode> suggest(final RelNode input, final Configuration configuration) { | ||
CommonTableExpressionRegistry localRegistry = new CommonTableExpressionRegistry(); | ||
HepProgram ruleProgram = new HepProgramBuilder() | ||
.addRuleInstance(CommonRelSubExprRegisterRule.JOIN) | ||
.addRuleInstance(CommonRelSubExprRegisterRule.AGGREGATE) | ||
.addRuleInstance(CommonRelSubExprRegisterRule.FILTER) | ||
.addRuleInstance(CommonRelSubExprRegisterRule.PROJECT) | ||
.build(); | ||
HepPlanner planner = new HepPlanner(ruleProgram, Contexts.of(localRegistry)); | ||
planner.setRoot(input); | ||
planner.findBestExp(); | ||
RelMetadataQuery mq = input.getCluster().getMetadataQuery(); | ||
Comparator<RelNode> rowCountCmp = Comparator.comparing(mq::getRowCount).reversed(); | ||
Comparator<RelNode> rowSizeCmp = Comparator.comparing(mq::getAverageRowSize).reversed(); | ||
// Criterion based sorting to ensure deterministic ordering of results. | ||
return localRegistry.entries() | ||
.sorted(Comparator.comparing(HiveCalciteUtil::countNodes).thenComparing(rowCountCmp).thenComparing(rowSizeCmp)) | ||
.collect(Collectors.toList()); | ||
} | ||
|
||
} |
44 changes: 44 additions & 0 deletions
44
...java/org/apache/hadoop/hive/ql/optimizer/calcite/CommonTableExpressionPrintSuggester.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one or more | ||
* contributor license agreements. See the NOTICE file distributed with | ||
* this work for additional information regarding copyright ownership. | ||
* The ASF licenses this file to you under the Apache License, Version 2.0 | ||
* (the "License"); you may not use this file except in compliance with | ||
* the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
package org.apache.hadoop.hive.ql.optimizer.calcite; | ||
|
||
import org.apache.calcite.plan.RelOptUtil; | ||
import org.apache.calcite.rel.RelNode; | ||
import org.apache.hadoop.conf.Configuration; | ||
import org.apache.hadoop.hive.common.classification.InterfaceAudience; | ||
import org.apache.hadoop.hive.ql.session.SessionState; | ||
|
||
import java.util.List; | ||
|
||
/** | ||
* Suggester for printing common table expressions to the Session's console. | ||
*/ | ||
@InterfaceAudience.Private | ||
public class CommonTableExpressionPrintSuggester implements CommonTableExpressionSuggester { | ||
|
||
private final CommonTableExpressionSuggester internal = new CommonTableExpressionIdentitySuggester(); | ||
|
||
@Override | ||
public List<RelNode> suggest(final RelNode input, final Configuration configuration) { | ||
List<RelNode> result = internal.suggest(input, configuration); | ||
for (int i = 0; i < result.size(); i++) { | ||
SessionState.getConsole().printInfo("CTE Suggestion " + i + ":" + RelOptUtil.toString(result.get(i)), false); | ||
} | ||
return result; | ||
} | ||
|
||
} |
54 changes: 54 additions & 0 deletions
54
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CommonTableExpressionRegistry.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,54 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one or more | ||
* contributor license agreements. See the NOTICE file distributed with | ||
* this work for additional information regarding copyright ownership. | ||
* The ASF licenses this file to you under the Apache License, Version 2.0 | ||
* (the "License"); you may not use this file except in compliance with | ||
* the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
package org.apache.hadoop.hive.ql.optimizer.calcite; | ||
|
||
import org.apache.calcite.rel.RelNode; | ||
import org.apache.hadoop.hive.common.classification.InterfaceAudience; | ||
|
||
import java.util.HashMap; | ||
import java.util.Map; | ||
import java.util.stream.Stream; | ||
|
||
/** | ||
* A registry of common table expressions for a given query. | ||
* <p>The registry is only meant to hold common expressions for a single query.</p> | ||
* <p>The class is not thread-safe.</p> | ||
*/ | ||
@InterfaceAudience.Private | ||
public final class CommonTableExpressionRegistry { | ||
/** | ||
* A unique collection of common table expressions. | ||
* <p>The expressions may contain internal planning concepts such as {@link org.apache.calcite.plan.hep.HepRelVertex}. | ||
* </p> | ||
*/ | ||
private final Map<String, RelNode> ctes = new HashMap<>(); | ||
|
||
/** | ||
* Adds the specified common table expression to this registry. | ||
* @param cte common table expression to be added to the registry. | ||
*/ | ||
public void add(RelNode cte) { | ||
this.ctes.put(cte.getDigest(), cte); | ||
} | ||
|
||
/** | ||
* @return a stream with all common table expression entries | ||
*/ | ||
public Stream<RelNode> entries() { | ||
return ctes.values().stream().map(HiveCalciteUtil::stripHepVertices); | ||
} | ||
} |
48 changes: 48 additions & 0 deletions
48
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CommonTableExpressionSuggester.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one or more | ||
* contributor license agreements. See the NOTICE file distributed with | ||
* this work for additional information regarding copyright ownership. | ||
* The ASF licenses this file to you under the Apache License, Version 2.0 | ||
* (the "License"); you may not use this file except in compliance with | ||
* the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
package org.apache.hadoop.hive.ql.optimizer.calcite; | ||
|
||
import org.apache.calcite.rel.RelNode; | ||
import org.apache.hadoop.conf.Configuration; | ||
import org.apache.hadoop.hive.common.classification.InterfaceAudience; | ||
import org.apache.hadoop.hive.common.classification.InterfaceStability; | ||
|
||
import java.util.List; | ||
|
||
/** | ||
* Suggester for finding and returning interesting expressions that appear more than once in a query. The notion of | ||
* "interesting" is specific to the actual implementation of this interface. | ||
* <p>In some cases the interesting expressions may be readily available in the input query while in others they could be | ||
* revealed after applying various transformations and algebraic equivalences.</p> | ||
* <p>The final decision about using (or not) the suggestions provided by this class lies to the query optimizer. For | ||
* various reasons (e.g, incorrect or expensive expressions) the optimizer may choose to reject/ignore the results | ||
* provided by this class.</p> | ||
* <p>Implementations of this interface must have a public no argument constructor since they are instantiated via reflection. | ||
* The class is instantiated once for each query under compilation and then thrown away.</p> | ||
* <p>The interface is experimental and subject to change without notice.</p> | ||
*/ | ||
@InterfaceAudience.Public | ||
@InterfaceStability.Unstable | ||
public interface CommonTableExpressionSuggester { | ||
/** | ||
* Suggests interesting expressions for the specified query and configuration. | ||
* @param input a relational expression representing the query under compilation. | ||
* @param configuration the configuration to use for identifying and returning interesting expressions. | ||
* @return a list with interesting expressions for the specified query | ||
*/ | ||
List<RelNode> suggest(RelNode input, Configuration configuration); | ||
} |
43 changes: 43 additions & 0 deletions
43
...va/org/apache/hadoop/hive/ql/optimizer/calcite/CommonTableExpressionSuggesterFactory.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
/* | ||
* Licensed to the Apache Software Foundation (ASF) under one or more | ||
* contributor license agreements. See the NOTICE file distributed with | ||
* this work for additional information regarding copyright ownership. | ||
* The ASF licenses this file to you under the Apache License, Version 2.0 | ||
* (the "License"); you may not use this file except in compliance with | ||
* the License. You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
package org.apache.hadoop.hive.ql.optimizer.calcite; | ||
|
||
import org.apache.hadoop.hive.common.classification.InterfaceAudience; | ||
import org.apache.hadoop.hive.conf.HiveConf; | ||
|
||
import java.util.Collections; | ||
|
||
@InterfaceAudience.Private | ||
public final class CommonTableExpressionSuggesterFactory { | ||
|
||
public static CommonTableExpressionSuggester create(HiveConf configuration) { | ||
String suggestName = configuration.getVar(HiveConf.ConfVars.HIVE_CTE_SUGGESTER_CLASS); | ||
if (suggestName == null || suggestName.isEmpty()) { | ||
return (query, conf) -> Collections.emptyList(); | ||
} | ||
try { | ||
Class<?> suggesterClass = Class.forName(suggestName); | ||
if (CommonTableExpressionSuggester.class.isAssignableFrom(suggesterClass)) { | ||
return (CommonTableExpressionSuggester) suggesterClass.newInstance(); | ||
} | ||
throw new IllegalArgumentException( | ||
suggesterClass.getSimpleName() + " must implement " + CommonTableExpressionSuggester.class.getSimpleName()); | ||
} catch (ClassNotFoundException | InstantiationException | IllegalAccessException e) { | ||
throw new RuntimeException(e); | ||
} | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently, all CTEs are weighted equally. A potentially future enhancement could be one where there is weight/rank associated with it which would feed into the optimizer decisions. This only matters if we want to restrict the number of CTEs to be considered (for shorter compilation time).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Weighting and ranking CTEs definitely makes sense and I see this as a responsibility of the
CommonTableExpressionSuggester
; concrete implementations of the interface could trade off speed for quality or the opposite.Furthermore, the
CommonTableExpressionSuggester
interface currently accepts aConfiguration
object so it is technically possible to pass arbitrary user-specified properties all the way to the suggester (e.g.,set mysupersuggester.cte.limit=5
) and fine tune specific aspects of its behavior.