Fix: Long IN-lists causes crash

[Swiddis]

Description

Per #1469, very large lists of expressions in IN clauses cause StackOverflowErrors. Avoiding all recursion here is nontrivial because DSL.or can only take two arguments (and is internally implemented as a LUT for performance). This PR instead rewrites the method to have the recursion amount grow logarithmically, by building the or tree as a balanced tree. This avoids hitting any recursion limits unless there's well over 2⁵¹² items in the list.

Implementing something like DSL.any is on the table for future work if we need to optimize further, but it's more complex than this fix and it's not clear that this is a hot path.

Related Issues

Resolves #1469

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dai-chen]

Thanks for the fix. In original issue, one of the possible solution is

"Compile IN expression to field = ANY(ARRAY(values)) which maybe optimal at both compile and runtime"

Could you compare it with current approach?

[Swiddis]

I wrote it in the description

Implementing something like DSL.any is on the table for future work if we need to optimize further, but it's more complex than this fix and it's not clear that this is a hot path.

It would probably be better than this, but it looks like we don't already have a DSL.any method available so I'd need to implement one. If you can give pointers on where to start, I can look into it?

I know from the or LUT that we pretty much just need to traverse the hierarchy of False -> Missing -> Null -> True when implementing it, just not sure where I'd register the function exactly...

[LantaoJin]

Will this improving be a throw away work in future? This issue only happens in v2. And with calcite, the query | where age in (10, 40.0, 100) will pushdown as

  "query": {
    "terms": {
      "age": [
        10,
        40,
        100
      ],
      "boost": 1
    }
  }

instead of following in v2:

  "query": {
    "bool": {
      "should": [
        {
          "term": {
            "age": {
              "value": 10,
              "boost": 1
            }
          }
        },
        {
          "bool": {
            "should": [
              {
                "term": {
                  "age": {
                    "value": 40,
                    "boost": 1
                  }
                }
              },
              {
                "term": {
                  "age": {
                    "value": 100,
                    "boost": 1
                  }
                }
              }
            ],
            "adjust_pure_negative": true,
            "boost": 1
          }
        }
      ],
      "adjust_pure_negative": true,
      "boost": 1
    }
  }

+1 for Compile IN expression to field = ANY(ARRAY(values)) if we really want to fix it in v2. But performance improving in v2 could be a low priority task since from 3.1.0+, the calcite engine may be enabled by default.