Skip to content

GH-35815: [C++][Compute] Support dictionary inputs in UTF-8 trim#50501

Open
LarryHu0217 wants to merge 3 commits into
apache:mainfrom
LarryHu0217:codex/arrow-dictionary-utf8-trim-35815
Open

GH-35815: [C++][Compute] Support dictionary inputs in UTF-8 trim#50501
LarryHu0217 wants to merge 3 commits into
apache:mainfrom
LarryHu0217:codex/arrow-dictionary-utf8-trim-35815

Conversation

@LarryHu0217

@LarryHu0217 LarryHu0217 commented Jul 13, 2026

Copy link
Copy Markdown

Rationale for this change

Pure unary scalar functions produce the same result for repeated input values. For dictionary-encoded input, running the function against every logical row is therefore unnecessary: the function can run once against the dictionary values and the original indices can expand the result.

What changes are included in this PR?

  • Add a generic dictionary adapter for fixed-arity, pure unary ScalarFunctions.
  • Dispatch the underlying function against the dictionary value type, execute it on the dictionary values, then expand the transformed values with the original indices.
  • Only advertise the adapter when the dictionary value type has a matching underlying kernel; impure unary functions remain unsupported.
  • Add generic coverage for type-changing output, unique-value execution, unsupported value types, and impure functions.
  • Keep issue-specific regression coverage for utf8_trim, utf8_ltrim, and utf8_rtrim with dictionary-encoded string and large-string inputs, including nulls.

Are these changes tested?

  • arrow-compute-internals-test --gtest_brief=1 (73 passed)
  • arrow-compute-scalar-type-test --gtest_brief=1 (344 passed, 1 pre-existing disabled test)
  • C++ Debug build with warnings treated as errors
  • clang-format --dry-run --Werror on all touched C++ files

Are there any user-facing changes?

Yes. Pure unary scalar functions can now accept dictionary-encoded inputs whenever their dictionary value type has a matching kernel. Results retain the function's existing decoded output type, so no public API signature changes are introduced.

AI-assisted development

OpenAI Codex assisted with reproducing the issue, implementing the patch, and drafting the regression tests. The change was checked against Arrow's scalar execution and dictionary dispatch paths and validated with the commands above.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #35815 has been automatically assigned in GitHub to PR creator.

@github-actions github-actions Bot added awaiting review Awaiting review Component: C++ and removed awaiting review Awaiting review labels Jul 13, 2026
@LarryHu0217
LarryHu0217 marked this pull request as ready for review July 16, 2026 01:15
@LarryHu0217
LarryHu0217 requested a review from pitrou as a code owner July 16, 2026 01:15
@github-actions github-actions Bot added the awaiting review Awaiting review label Jul 16, 2026
@pitrou

pitrou commented Jul 22, 2026

Copy link
Copy Markdown
Member
  • Decode dictionary input types during dispatch for utf8_trim, utf8_ltrim, and utf8_rtrim.

This is wasteful, isn't it? The scalar function could just be applied to the dictionary values.

Also I don't understand why we would only tackle these three specific functions, while this could apply to all scalar unary functions.

cc @zanmato1984 for opinions.

@LarryHu0217

Copy link
Copy Markdown
Author

Thanks, addressed both points in d2f3763. The trim-specific dispatch subclass is gone. The new adapter applies generically to fixed-arity pure unary ScalarFunctions when the dictionary value type has a matching kernel, executes the function against the dictionary values, and expands the result with the original indices while retaining the existing decoded output type. I added generic tests for type-changing output, unique-value execution, unsupported value types, and impure functions, plus the trim regressions. Local results: 73/73 compute-internals tests and 344/344 scalar-type tests passed.

@github-actions

Copy link
Copy Markdown

⚠️ GitHub issue #35815 has been automatically assigned in GitHub to PR creator.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants