fix: preserve null semantics in expression simplification

[fzlzjerry]

Which issue does this PR close?

• Closes Expression simplifications may produce incorrect results with NULLs #24246.

Rationale for this change

Several expression simplifications erase SQL NULL semantics. For nullable inputs, identities such as power(a, 0), log(a, 1), and i XOR i can return constants instead of NULL. Set algebra over IN expressions and the array_has to IN rewrite can similarly change NULL or false results.

What changes are included in this PR?

• Apply the affected logarithm, power, and XOR identities only when the operand being eliminated is non-nullable.
• Keep IN/NOT IN set operations unchanged when list items may be NULL, and preserve the nullable branch when an otherwise safe operation becomes a constant.
• Avoid rewriting array_has to IN when a nullable array element would change the result, while retaining the safe same-expression case.
• Add unit and SQL logic regression coverage for the reported expressions and their query plans.

Are these changes tested?

Yes. The following checks pass:

• ./dev/rust_lint.sh
• cargo test --profile=ci -p datafusion-functions -p datafusion-functions-nested -p datafusion-optimizer --lib
• cargo test --profile=ci --test sqllogictests
• RUST_BACKTRACE=1 prlimit --nofile=65536:65536 -- cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryption

Are there any user-facing changes?

Yes. Expressions involving nullable inputs now retain their SQL three-valued-logic results instead of being simplified to incorrect constants. There are no public API changes.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[AdamGS]

I'll try and review it later today

[codecov-commenter]

Codecov Report

❌ Patch coverage is 92.19512% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.01%. Comparing base (a05388e) to head (ea308e2).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...imizer/src/simplify_expressions/expr_simplifier.rs	92.22%	3 Missing and 4 partials ⚠️
datafusion/functions-nested/src/array_has.rs	94.52%	3 Missing and 1 partial ⚠️
datafusion/functions/src/math/log.rs	85.00%	1 Missing and 2 partials ⚠️
datafusion/functions/src/math/power.rs	90.90%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #24258    +/-   ##
========================================
  Coverage   81.01%   81.01%            
========================================
  Files        1106     1106            
  Lines      383974   384228   +254     
  Branches   383974   384228   +254     
========================================
+ Hits       311085   311297   +212     
- Misses      54554    54577    +23     
- Partials    18335    18354    +19     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[neilconway]

FYI there is also #24247 and #24248

[AdamGS]

@neilconway thanks for the heads up, I'll try and do a pass on all three tonight/tomorrow morning (UK time).

[fzlzjerry]

Thanks for flagging these. I compared the scopes:

• fix: preserve NULL semantics in log and power simplification #24247 overlaps the log / power fixes.
• fix: preserve NULL semantics in bitwise xor simplification #24248 overlaps the XOR cancellation fix.
• This PR additionally covers the IN / NOT IN set operations and the array_has to IN rewrite from Expression simplifications may produce incorrect results with NULLs #24246.

I'll keep the current comparison available for the review pass, then promptly rebase and drop whichever overlapping implementation the maintainers decide not to use.