HIVE-28363: Improve heuristics of FilterStatsRule without column stats

[okumin]

What changes were proposed in this pull request?

Make FilterStatsRule reduce the number of filtered rows by half when # of distinct values is empty.
https://issues.apache.org/jira/browse/HIVE-28363

Why are the changes needed?

Simply, col IN (5) makes the estimated amount half, col IN (5, 10) keeps 100% of stats, col IN (5, 10, 15) keeps 100% of stats, and so on. I expect col IN ({arbitrary number of constants}) to filter out rows to some extent in almost all cases.

FilterStatsRule roughly estimates the number of rows filtered by IN to be {Original # of rows} * {1 / cardinality} * {# of values in IN}. The second term is estimated as 0.5 when column stats are unavailable. So, it always returns the original number when IN retains two constant values like col IN (1, 3).

Maybe, FilterStatsRule had behaved in the same way as this PR before, but this change slightly changed the formula to cover a special case. We will likely prefer the original formula when columns stats are not available.

Does this PR introduce any user-facing change?

No.

Is the change a dependency upgrade?

No.

How was this patch tested?

CREATE TABLE users (id INT);
INSERT INTO users VALUES (1), (2), (3), (4), (5), (6), (7), (8), (9), (10);
set hive.fetch.task.conversion=none;
set hive.stats.fetch.column.stats=false;
EXPLAIN SELECT * FROM users WHERE id IN (1);
EXPLAIN SELECT * FROM users WHERE id IN (1, 2);

[sonarcloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[kgyrtkirk]

I think you could possibly introduce a hiveconf key for setting this parameter as well; I guess a default of .1 would be even better...

[okumin]

Can we use this property to tune the factor? It could not directly support your original intention, but it will likely work as well or even better.

[kgyrtkirk]

sure you could use that as well - that's also better than burning in.5

[okumin]

@kgyrtkirk Can I keep the current revision? The parameter is multiplied at the end of the overall calculation. I guess it is more desirable than multiplying the arbitrary parameter in this loop. If we use 0.1 in this loop, and if the size of columnStats is 3, the final factor will be 0.1 * 0.1 * 0.1. Maybe, it is overkilling

[okumin]

@kgyrtkirk Would you mind accepting the proposal and merging this PR? Thanks.

[okumin]

@kgyrtkirk Can we merge this one or do I need something to fix? Thanks!

[sonarcloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[okumin]

I rebased this branch on the current master yesterday

[zhangbutao]

LGTM +1
Based on the provided&changed test cases, i think this change is reasonable, especially for the external tables without column stats.

IMO, statistics are only a rough estimate, and we can not make it perfect. So we can get this change in first, and then continue to optimize it.

BTW, we can also consider to how to evaluate Iceberg table stats to optimize the queries in the StatsRulesProcFactory. This can be done in the future work.

Hi, @kgyrtkirk, do you have any other comments? :)
Thanks.

[zhangbutao]

@kgyrtkirk I just merged the change. If you have further comments. we can optimize it later. :)
Thanks.

[okumin]

Thank you, both!