Skip to content

IN LIST: treat signed zeros as equal - #25186

Draft
geoffreyclaude wants to merge 1 commit into
apache:mainfrom
geoffreyclaude:codex/in-list-signed-zero
Draft

IN LIST: treat signed zeros as equal#25186
geoffreyclaude wants to merge 1 commit into
apache:mainfrom
geoffreyclaude:codex/in-list-signed-zero

Conversation

@geoffreyclaude

@geoffreyclaude geoffreyclaude commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

None

Rationale for this change

DataFusion equality treats +0.0 and -0.0 as equal, but the optimized IN filters compare or hash floating-point bit patterns. Because the two zeros have different encodings, an IN predicate could therefore return a different result depending on whether the optimizer retained it or rewrote it into equality comparisons.

This PR makes top-level Float16, Float32, and Float64 IN predicates follow the same signed-zero equality as ordinary comparisons. Static filters materialize both zero encodings once while building the list, keeping lookup on the row path branch-free. Non-static lists normalize evaluated floating operands once per batch. Distinct NaN payloads remain distinct, preserving the existing bit-equality behavior for NaNs.

What changes are included in this PR?

  • Adds both signed-zero encodings to branchless floating filters when either zero occurs in the static list.
  • Applies the same rule to the Float16 bitmap and Float32 / Float64 hash-set filters.
  • Extends the shared zero normalizer to dictionary-wrapped floating arrays and scalar values for non-static lists and equality rewrites.
  • Preserves null behavior, dictionary inputs, IN, and NOT IN.

Are these changes tested?

The SQL logic tests cover both signed-zero directions for Float16, Float32, and Float64; comparison-rewritten, static-filter, and non-static-list paths; dictionary-encoded Float64; and both IN and NOT IN. The added cases fail on the parent commit and pass with this fix.

Focused validation:

  • cargo test -p datafusion-common normalize_float_zero
  • cargo test -p datafusion-physical-expr in_list (88 tests)
  • cargo test -p datafusion-sqllogictest --test sqllogictests -- negative_zero

Are there any user-facing changes?

Yes. Floating-point IN and NOT IN now treat +0.0 and -0.0 as equal, consistently with ordinary DataFusion equality. There are no public API changes.

@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) common Related to common crate labels Sep 11, 2026
@geoffreyclaude
geoffreyclaude force-pushed the codex/in-list-signed-zero branch 5 times, most recently from 12098fc to 474474f Compare September 11, 2026 13:58
@codecov-commenter

codecov-commenter commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.50000% with 37 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.92%. Comparing base (224cc56) to head (eaee11e).
⚠️ Report is 5 commits behind head on main.

Files with missing lines Patch % Lines
...atafusion/physical-expr/src/expressions/in_list.rs 80.90% 4 Missing and 17 partials ⚠️
datafusion/common/src/utils/mod.rs 86.48% 8 Missing and 2 partials ⚠️
...l-expr/src/expressions/in_list/primitive_filter.rs 94.73% 0 Missing and 4 partials ⚠️
...-expr/src/expressions/in_list/branchless_filter.rs 94.44% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             main   #25186    +/-   ##
========================================
  Coverage   81.91%   81.92%            
========================================
  Files        1132     1132            
  Lines      421314   421577   +263     
  Branches   421314   421577   +263     
========================================
+ Hits       345135   345365   +230     
- Misses      55771    55783    +12     
- Partials    20408    20429    +21     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@geoffreyclaude
geoffreyclaude force-pushed the codex/in-list-signed-zero branch 2 times, most recently from e0f44e0 to aa5a037 Compare September 11, 2026 14:36
let normalized: Float32Array =
arr.unary(|v| if v.to_bits() << 1 == 0 { 0.0_f32 } else { v });
let normalized: Float32Array = arr.unary(|v| {
if v.to_bits() == NEG_ZERO_F32_BITS {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only NEG_ZERO_F32_BITS needs a change, so compare directly against it

let normalized: Float64Array =
arr.unary(|v| if v.to_bits() << 1 == 0 { 0.0_f64 } else { v });
let normalized: Float64Array = arr.unary(|v| {
if v.to_bits() == NEG_ZERO_F64_BITS {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only NEG_ZERO_F64_BITS needs a change, so compare directly against it

@geoffreyclaude
geoffreyclaude force-pushed the codex/in-list-signed-zero branch from aa5a037 to eaee11e Compare September 11, 2026 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common Related to common crate physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants