Fix: spark bit_count function #18322
Conversation
| } | ||
|
|
||
| // Here’s the equivalent Rust implementation of the bitCount function (similar to Apache Spark's bitCount for LongType) | ||
| fn bit_count(i: i64) -> i32 { |
There was a problem hiding this comment.
Could we enhance the comment here with this link + potentially link to original source code this actual function is copied from?
There was a problem hiding this comment.
Thanks, added
There was a problem hiding this comment.
Where was the actual bitcount implementation code pulled from though?
There was a problem hiding this comment.
Added additional link to Java implementation of bitCount function
| let value_array = value_array[0].as_ref(); | ||
| match value_array.data_type() { | ||
| DataType::Int8 => { | ||
| DataType::Int8 | DataType::Boolean => { |
There was a problem hiding this comment.
Spark supports only signed int types
There was a problem hiding this comment.
This comment seems misplaced as the code adds support for boolean 🤔
There was a problem hiding this comment.
I also think this code will now panic if you pass in a Boolean array
There was a problem hiding this comment.
I wanted to note that Spark only supports signed integer and boolean types as input.
There was a problem hiding this comment.
Added an additional test for BooleanArray.
3 participants
Which issue does this PR close?
Closes #18225
Rationale for this change
After adding the bit_count function in Comet, we got different results from Spark. (apache/datafusion-comet#2553)
Are these changes tested?
Tested with existing unit tests