Skip to content

feat(spark): add to_binary and try_to_binary - #24266

Open
arsathS wants to merge 2 commits into
apache:mainfrom
arsathS:spark-to-binary
Open

feat(spark): add to_binary and try_to_binary#24266
arsathS wants to merge 2 commits into
apache:mainfrom
arsathS:spark-to-binary

Conversation

@arsathS

@arsathS arsathS commented Aug 11, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

Spark's to_binary(str[, fmt]) converts a string to binary, choosing how based
on fmt: hex, utf-8/utf8, or base64. try_to_binary is the same
function except that input it cannot convert yields NULL instead of raising an
error.

Neither is available in the datafusion-spark crate, and no core DataFusion
function can stand in for them:

  • core decode accepts only base64, base64pad and hex, so there is no
    way to ask it for Spark's utf-8 conversion
  • core decode always raises on input it cannot convert, so there is nothing
    to build try_to_binary's NULL-returning behaviour from

What changes are included in this PR?

  • New SparkToBinary and SparkTryToBinary UDFs in
    datafusion/spark/src/function/string/to_binary.rs
    • to_binary(str[, fmt]) where fmt is a case-insensitive literal of
      hex, utf-8, utf8 or base64, defaulting to hex
    • hex decodes two characters per byte, left-padding an odd-length input
      with 0, matching Unhex
    • utf-8 / utf8 returns the string's own UTF-8 bytes
    • base64 matches Java's MIME decoder, which Spark uses: the standard
      alphabet, padding optional, and the unused trailing bits of a short final
      group ignored
    • A NULL value or a NULL fmt yields NULL in both functions
    • to_binary raises on a malformed value or an unrecognised fmt;
      try_to_binary returns NULL for both, matching nullOnInvalidFormat
    • fmt must be foldable, as Spark requires
  • unhex_scalar is made pub(crate) and reused for the hex path rather
    than duplicating the decoder
  • Registered in mod.rs (make_udf_function!, export_functions!,
    functions())
  • Filled in the commented-out SLT scaffolds for both functions

Are these changes tested?

Yes.

  • 10 unit tests in to_binary.rs (each format, hex as the default, empty
    input, NULL value and NULL format, invalid value, invalid format, column
    input, and a column where one row is invalid)
  • 15 SLT assertions across spark/string/to_binary.slt and
    spark/string/try_to_binary.slt

Are there any user-facing changes?

No. These are new functions in the datafusion-spark crate only.

Implements `to_binary(str[, fmt])` and `try_to_binary(str[, fmt])` for the
datafusion-spark crate. `fmt` is a case-insensitive literal of `hex`,
`utf-8`, `utf8` or `base64`, defaulting to `hex`.

The conversion is done in the kernel rather than by delegating to existing
functions: Spark rewrites `to_binary` onto `Unhex`/`Encode`/`UnBase64` with
`failOnError = true`, but the equivalents here disagree on which inputs error
and which return NULL, so a single `fail_on_error` flag drives both variants.
`unbase64` also has no kernel to call -- it exists only as a `simplify()`
rewrite -- and core's base64 decoder is private.

Hex decoding reuses the existing `unhex_scalar`, which is made `pub(crate)`
rather than duplicated.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) spark labels Aug 11, 2026
@arsathS

arsathS commented Aug 11, 2026

Copy link
Copy Markdown
Author

cc @Jefffrey This is my first time contribution. Can you please take a look?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

spark sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant