[SPARK-58553][PS][FOLLOWUP] Match fmax/fmin signed-zero tie-break to installed NumPy

[Spenserrrr]

What changes were proposed in this pull request?

SPARK-58553 replaced the pandas_udf-based np.fmax / np.fmin implementations in the pandas API on Spark with native F.greatest / F.least expressions. When the two operands are equal (for example +0.0 and -0.0), the native mapping breaks the tie by returning the first operand.

NumPy changed this signed-zero tie-break at 2.3.0: >= 2.3.0 returns the first operand, while older versions return the second. The native mapping therefore matches NumPy >= 2.3.0 but disagrees with older versions on the sign of a ±0.0 result.

This PR selects the tie operand based on the installed NumPy version so the result matches np.fmax / np.fmin on that NumPy: return the first operand on >= 2.3.0, the second on older versions. The implementation stays fully native (F.greatest / F.least); only which operand is returned on a signed-zero tie differs by version.

Why not restore the original pandas_udf fallback for old NumPy? The original UDF matched the installed NumPy automatically (it calls np.fmax in a Python worker), so restoring it for < 2.3.0 would also be correct. But that reintroduces the per-batch JVM <-> Python round trip that SPARK-58553 removed, losing the performance and optimizer benefits for those users. Since the signed-zero tie is the only cross-version difference (verified exhaustively over every combination of {-inf, -2, -1, -0.0, +0.0, 1, 2, inf, nan} from NumPy 1.23.2 through 2.4.1 — the tie-break flips at 2.3.0 and nothing else changes), and NumPy < 2.3.0 is a frozen release range, selecting the matching tie operand keeps the native fast path while producing results identical to np.fmax / np.fmin.

Scope: fmax / fmin is the only affected function. The full test_numpy_compat.py suite (18 tests, including the generic mapping sweeps and every other SPARK-58532 conversion — fmod, ldexp, heaviside, reciprocal, float_power, bitwise shifts, signbit, etc.) passes on the minimum dependencies; the signed-zero tie in fmax / fmin is the only version-sensitive behavior.

Why are the changes needed?

The scheduled "Build / Python-only (Minimum dependencies of PySpark)" build (NumPy 1.23.2) fails pyspark.pandas.tests.test_numpy_compat NumPyCompatTests.test_np_fmax_fmin. The test asserts the sign bit of the result via np.signbit, and on the two ±0.0 tie rows the native mapping (first operand) disagrees with the reference computed from the installed NumPy (second operand on 1.23.2). Regular CI runs a newer NumPy (>= 2.3.0), where the native choice matches, which is why the original change passed pre-merge CI and the failure only surfaced in the minimum-dependency build.

Does this PR introduce any user-facing change?

No. There is no change relative to any released Spark version (the released implementation used the pandas_udf, which already matched the installed NumPy). This aligns the unreleased native implementation from SPARK-58553 with np.fmax / np.fmin on NumPy < 2.3.0. The numeric value is unchanged in all cases (+0.0 and -0.0 are numerically equal); only the sign bit of a zero result on a ±0.0 tie is corrected to match the installed NumPy.

How was this patch tested?

• pyspark.pandas.tests.test_numpy_compat.NumPyCompatTests.test_np_fmax_fmin and the full NumPyCompatTests suite (18 tests) pass on NumPy 2.4.1 and in a minimum-dependency environment (NumPy 1.23.2, pandas 2.2.0, pyarrow 18.0.0) that reproduces the scheduled build.
• Confirmed the failure reproduces on NumPy 1.23.2 without this change and is resolved with it.
• Verified across installed NumPy wheels (1.23.2 through 2.4.1) that the signed-zero ±0.0 tie is the only fmax / fmin behavior that differs between versions, and that the tie-break flips at exactly 2.3.0.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

[Spenserrrr]

Hi @zhengruifeng! This is a PR to fix min-deps build failure on SPARK-58553. Could you take a look when you have time? Thanks! cc @Yicong-Huang

[zhengruifeng]

Merge Summary:

• merged into master 614deed
• merged into branch-4.x 0710f87

Posted by merge_spark_pr.py