[SPARK-58428][SQL] Fix optimizer hang in DSv2 expression pushdown

[AbhinavBattu]

What changes were proposed in this pull request?

V2ExpressionBuilder.generateExpression handles a context independent foldable
expression by constant folding it and recursing on the result, so that the folded
value is translated by the Literal case:

case _ if expr.contextIndependentFoldable && <datasourceV2ExprFolding> =>
  val constantExpr = ConstantFolding.constantFolding(expr)
  generateExpression(constantExpr, isPredicate)

That assumes constant folding returns a literal. It does not. ConstantFolding
returns an expression unchanged when it carries the FAILED_TO_EVALUATE tag, which
is set when evaluation failed inside a conditional branch so that the error is not
raised at planning time for a branch that may never be reached. The recursive call
then re-enters the same case with the same expression and never makes progress.
Because the call is in tail position it loops instead of raising
StackOverflowError, so the symptom is a query that never returns.

This PR recurses only when folding actually changed the expression.

This follows SPARK-50380, which made ReorderAssociativeOperator stop assuming that
a foldable expression folds to a literal.

The guard is on the folding result rather than on the tag, so it also covers any
future case where constantFolding returns its input unchanged.

Why are the changes needed?

With default settings, a pushdown eligible query against a DSv2 source hangs when a
filter contains an expression that fails to evaluate inside a conditional branch:

SET spark.sql.catalog.d=org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog;
SET spark.sql.catalog.d.url=jdbc:derby:memory:v2loopdb;
SET spark.sql.catalog.d.driver=org.apache.derby.jdbc.EmbeddedDriver;
SET spark.sql.catalog.d.create=true;

CREATE NAMESPACE IF NOT EXISTS d.test;
CREATE TABLE d.test.t (c INT);

SELECT count(*) FROM d.test.t WHERE c = 1;              -- control: 0
SELECT * FROM d.test.t WHERE coalesce(c, 1 div 0) = 1;  -- never returns

Four conditions have to hold, which is why this was not hit earlier:

• ANSI mode, so 1 div 0 throws while the planner folds it
• the failing expression inside a conditional, so the error is deferred and tagged
  rather than raised
• a DSv2 source that pushes predicates, since V2ExpressionBuilder is only used on
  that path
• spark.sql.optimizer.datasourceV2ExprFolding, which defaults to true and was
  added in 4.1.0

Setting spark.sql.optimizer.datasourceV2ExprFolding to false avoids the hang.

Does this PR introduce any user-facing change?

Yes. A query of the shape above previously hung and now completes. Such an
expression is no longer translated, so the predicate is not pushed to the source and
is evaluated by Spark instead. Query results are unchanged.

How was this patch tested?

Added a test to DataSourceV2StrategySuite, next to the existing
datasourceV2ExprFolding test. It builds coalesce(c, 1 div 0) = 1, asserts that
constant folding tagged the failing branch, and then asserts that translation
produces no V2 expression. The tag assertion is there so that the test cannot pass
for the wrong reason if the tagging behaviour changes.

Without this change the test does not terminate. The recursion is in tail position
and does not block, so it cannot be interrupted by a time limit, which is why the
test asserts the result instead of using failAfter. This matches existing tests
for similar issues, for example "SPARK-48843: Prevent infinite loop with
BindParameters" in ParametersSuite.

build/sbt 'sql/testOnly *DataSourceV2StrategySuite *JDBCV2Suite'

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5)

[uros-b]

Thank you @AbhinavBattu! Adding @gengliangwang to also take a look