[SPARK-57775][SQL][4.1] Return one row for GROUP BY GROUPING SETS (()) over empty input

[uros-b]

What changes were proposed in this pull request?

Note: this is a 4.1 backport of:

• [SPARK-57775][SQL] Return one row for GROUP BY GROUPING SETS (()) over empty input #56891 (master/4.x)
• [SPARK-57775][SQL][4.2] Return one row for GROUP BY GROUPING SETS (()) over empty input #57134 (4.2)

Lower the single-empty-grouping-set case to a global Aggregate (no grouping expressions, no Expand), so it returns one (grand total) row over empty input, matching the GROUP BY-less form and the SQL standard. grouping_id() folds to the constant 0, and grouping()/grouping_id() in HAVING/ORDER BY resolve against that constant.

Unlike the master/4.2 change, which lives in GroupingAnalyticsTransformer (introduced later by SPARK-54961), branch-4.1 has no such object: the grouping-analytics lowering is still inline in Analyzer.ResolveGroupingAnalytics, so the fix is re-implemented there. branch-4.1 also has no single-pass GroupingAnalyticsResolver (grouping analytics is not supported by the single-pass resolver on 4.1, guarded out in ResolverGuard), so the fix lands only in the legacy fixed-point analyzer.

Why are the changes needed?

GROUP BY GROUPING SETS (()) is a grand total, semantically identical to an aggregation with no GROUP BY clause. It was lowered to a grouped Aggregate over an Expand (grouping by spark_grouping_id), so over empty input it returned zero rows instead of one. The same defect affected the equivalent empty GROUP BY CUBE() and GROUP BY ROLLUP(), which also lower to a single empty grouping set.

Does this PR introduce any user-facing change?

Yes, the new (correct) behavior is gated by an internal SQL config,
spark.sql.analyzer.lowerEmptyGroupingSetToGlobalAggregate.enabled (default true). When set to false, lowering reverts to the legacy Expand-based form (zero rows over empty input). The flag gates both the transformer lowering and the grouping_id() resolution in HAVING/ORDER BY, so the off state reproduces pre-fix behavior identically.

How was this patch tested?

Tested via golden cases in grouping_set.sql (empty-input grand total, grouping_id() in SELECT/HAVING/ORDER BY, grouping() rejection, non-empty input), flag-off coverage in grouping_set_grand_total_disabled.sql, ResolveGroupingAnalyticsSuite, DataFrameAggregateSuite, and the Connect ClientE2ETestSuite.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Opus 4.8

[uros-b]

Waiting for CI.

[dongjoon-hyun]

Although this is a bug fix, I'd be more conservative when we deliver this to Spark 4.1.4, @uros-b . Instead of forcefully enforcing the behavior change, can we simply deliver this feature with spark.sql.analyzer.lowerEmptyGroupingSetToGlobalAggregate.enabled=false?

[dongjoon-hyun]

May I ask if you already deliver this bug fix to your company product in this way, @uros-b ?