[WIP] PartitionEliminationFilterIndexRule implementation

[apoorvedave1]

What is the context for this pull request?

In this PR we introduce FilterIndexRule for PartitionEliminationNonCoveringIndex. Rule Algorithm:

1. Identify if an index contains index columns which can improve point lookups or range queries on data source
2. Query the index using spark to identify list of data files which could satisfy the query
3. Redirect original query to this subset of data files instead of complete list of files.

• Tracking Issue: If you expect any subjective discussions around this pull request, please consider opening a tracking issue and link to the PR. Write N/A, if this pull request is self-contained.
• Parent Issue: Link to the issue that captures the overall plan. Write N/A, if this is a stand-alone pull request with a tracking issue OR self-contained pull request.
• Dependencies: Links to issues you depend on for this pull request to work. Write N/A, if no dependencies.
  • Issue 1
  • Issue 2

What changes were proposed in this pull request?

Does this PR introduce any user-facing change?

No

How was this patch tested?

[sezruby]

I thought we build an index of (value => file id list) instead of full scan of covering index.
Could you measure the perf of optimize phase with 1TB TPCH dataset?

[apoorvedave1]

this would not work if we want to merge Covering and partition elimination index.

[sezruby]

We cannot deliver this rule without a proper performance validation as it seems expensive. Could you run TPCH 1TB 100k chunk dataset and share the result?

1. w/o FEFilterIndexRule for comparison - explain time & query execution time
2. w/ PEFilterindexRule - explain time & query execution time