GH-35815: [C++][Compute] Support dictionary inputs in UTF-8 trim#50501
GH-35815: [C++][Compute] Support dictionary inputs in UTF-8 trim#50501LarryHu0217 wants to merge 3 commits into
Conversation
|
|
This is wasteful, isn't it? The scalar function could just be applied to the dictionary values. Also I don't understand why we would only tackle these three specific functions, while this could apply to all scalar unary functions. cc @zanmato1984 for opinions. |
|
Thanks, addressed both points in d2f3763. The trim-specific dispatch subclass is gone. The new adapter applies generically to fixed-arity pure unary ScalarFunctions when the dictionary value type has a matching kernel, executes the function against the dictionary values, and expands the result with the original indices while retaining the existing decoded output type. I added generic tests for type-changing output, unique-value execution, unsupported value types, and impure functions, plus the trim regressions. Local results: 73/73 compute-internals tests and 344/344 scalar-type tests passed. |
|
|
Rationale for this change
Pure unary scalar functions produce the same result for repeated input values. For dictionary-encoded input, running the function against every logical row is therefore unnecessary: the function can run once against the dictionary values and the original indices can expand the result.
What changes are included in this PR?
ScalarFunctions.utf8_trim,utf8_ltrim, andutf8_rtrimwith dictionary-encoded string and large-string inputs, including nulls.Are these changes tested?
arrow-compute-internals-test --gtest_brief=1(73 passed)arrow-compute-scalar-type-test --gtest_brief=1(344 passed, 1 pre-existing disabled test)clang-format --dry-run --Werroron all touched C++ filesAre there any user-facing changes?
Yes. Pure unary scalar functions can now accept dictionary-encoded inputs whenever their dictionary value type has a matching kernel. Results retain the function's existing decoded output type, so no public API signature changes are introduced.
AI-assisted development
OpenAI Codex assisted with reproducing the issue, implementing the patch, and drafting the regression tests. The change was checked against Arrow's scalar execution and dictionary dispatch paths and validated with the commands above.