[Python] Implement unification of null dictionaries

[alanhdu]

Describe the enhancement requested

Currently, if you try to do something like:

a = pa.array([None, None, None]).cast(pa.dictionary(pa.int8(), pa.null()))
b = pa.array([None, None, None]).cast(pa.dictionary(pa.int8(), pa.null()))

pa.chunked_array([a, b]).unify_dictionaries()

PyArrow will raise an exception like:

File pyarrow/table.pxi:1206, in pyarrow.lib.ChunkedArray.unify_dictionaries()

File pyarrow/error.pxi:154, in pyarrow.lib.pyarrow_internal_check_status()

File pyarrow/error.pxi:91, in pyarrow.lib.check_status()

ArrowNotImplementedError: Unification of null dictionaries is not implemented

Is there any way to just implement this? Manually calling .combine_chunks() seems to work fine, so the logic is clearly implemented somewhere.

a = pa.array([None, None, None]).cast(pa.dictionary(pa.int8(), pa.null()))
b = pa.array([None, None, None]).cast(pa.dictionary(pa.int8(), pa.null()))

pa.chunked_array([a, b]).combine_chunks()

I admit this might be a very niche feature (I am confused why I have feather files with these dictionary-encoded nulls, but I do and it'd be nice to be able to handle them).

Versions

I am testing this on Python 3.10 and pyarrow 15.

Component(s)

Python

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.

[alanhdu]

I checked and this is still a problem with PyArrow 22. I would personally prefer to keep this open, even if there is no volunteer to implement it (if someone is willing to give some pointers, I'd be happy to take a stab at it but I'm not familiar at all with the codebase).

[thisisnic]

This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.

[hlongmore]

This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.

So the comment by the author the day after the warning notice didn't count as activity, even though the warning notice said "Please remove the stale label or comment below" to avoid having it closed?

[thisisnic]

Probably just got picked up by accident, we can reopen it.

Will mark it as "needs champion" unless either of you are interested in submitting a PR?

[hlongmore]

IDK if this would help with implementing, but I was running into this issue because I have categorical data that I saved to parquet using pandas, with at least one of the categories as a partition column; then reading the parquet using dataframe = pd.read_parquet(filepath, engine='pyarrow', dtype_backend='pyarrow'). The read happens just fine, but when I try to e.g. create a mask or a column based on isna() and then try to use that to filter the dataframe, I get the above error. Poking around in the partition directories, I found that the partition directories for NA values for the categorical were written as column_name=__HIVE_DEFAULT_PARTITION__, and these values did not behave well on read (I couldn't even convert the column to string type to try to convert it to category after). I bulk renamed the directories with find . -type d | grep "HIVE_DEFAULT_PARTITION" | sed 'p;s/__HIVE_DEFAULT_PARTITION__/NA_unknown/' | xargs -n2 mv, reloaded the data, and was able to proceed just fine.

[alanhdu]

@thisisnic: I can try to send a PR, but I know almost no C++ and so I don't think it'd be much better than just asking Claude to do it xP. I'm happy to send a PR if it won't just spam you with more work.