feat: Improve fallback mechanism for ANSI mode

[andygrove]

Which issue does this PR close?

Closes #1974

Part of #313

Rationale for this change

Rather than fall back for all queries when ANSI mode is enabled, just fall back if the query is using expressions that are affected by ANSI mode.

What changes are included in this PR?

• Remove COMET_ANSI_MODE_ENABLED config and remove references to that in the Spark diffs since it is no longer required
• Add expression-specific ANSI fallback logic for all expressions that we have implemented that have an ANSI mode:
  • arithmetic expressions (Add, Subtract, Multiply, Divide, IntegralDivide, and Remainder)
  • aggregate expressions (Avg and Sum)
  • Round
  • Cast
• Update golden files for Spark 4
  • We now specify spark.comet.expression.allowIncompatible, so that sum and avg run natively. This had the side effect of allowing more operators to run natively than before,

How are these changes tested?

[codecov-commenter]

Codecov Report

❌ Patch coverage is 31.81818% with 30 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.97%. Comparing base (f09f8af) to head (16099a1).
⚠️ Report is 417 commits behind head on main.

Files with missing lines	Patch %	Lines
...main/scala/org/apache/comet/serde/aggregates.scala	27.27%	12 Missing and 4 partials ⚠️
...main/scala/org/apache/comet/serde/arithmetic.scala	38.09%	6 Missing and 7 partials ⚠️
...scala/org/apache/comet/expressions/CometCast.scala	0.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2211      +/-   ##
============================================
+ Coverage     56.12%   57.97%   +1.84%     
- Complexity      976     1275     +299     
============================================
  Files           119      143      +24     
  Lines         11743    13312    +1569     
  Branches       2251     2375     +124     
============================================
+ Hits           6591     7717    +1126     
- Misses         4012     4344     +332     
- Partials       1140     1251     +111     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[coderfender]

This is great @andygrove. Adding a comment to link PR #2136 which is created to implement ANSI mode for arithmetic operations.

[kazuyukitanimura]

When ANSI was not enabled, was this passing?
If so, is there a way to check whether ANSI is enabled and skip only when both comet and ansi are enabled?

[andygrove]

We were previously falling back to Spark for this test. This Spark test is specifically for testing ANSI errors from cast.

[andygrove]

I don't recall now why we were falling back to Spark so I will take another look

[andygrove]

For Spark 3.x we previously fell back to Spark for this test because we do not run the Spark SQL tests for 3.x with ENABLE_COMET_ANSI_MODE=true and therefore did not enable spark.comet.ansi.enabled.

[mbutrovich]

Thanks @andygrove. I think we should have a bigger discussion about defaults and tuning suggestions for ANSI mode.