[WIP][SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates#57223
Open
stevomitric wants to merge 2 commits into
Open
Conversation
…oercion in comparisons and predicates
Type coercion previously handled only microsecond TimestampType when one
operand of a comparison or predicate is a string: the string was cast to
TimestampType so subsecond digits compared exactly. Nanosecond timestamp
types (Timestamp{LTZ,NTZ}NanosType) had no such arm, so a string compared
against a nanos column fell through to generic handling and lost the 7th-9th
fractional digits.
This adds the missing non-ANSI arms, mirroring the existing microsecond
handling: two Equality arms in StringPromotionTypeCoercion cast the string to
the concrete nanos operand type, and two findCommonTypeForBinaryComparison
cases in TypeCoercion return that nanos type for the range path. The concrete
family and precision of the timestamp operand are preserved, the arms are
ordered so equality fires before the range path, and legacy
castDatetimeToString is honored (range promotes both sides to string, equality
still casts the string to nanos). ANSI type coercion is intentionally left
unchanged.
Tests: TypeCoercionSuite "binary comparison with string promotion" gains
nanos assertions (all precisions, both families, plus a legacy-mode block that
exercises the new arms); TimestampNanosWideningSuiteBase adds an end-to-end
string-operand test; and the timestamp-{ltz,ntz}-nanos golden files add
equality/range/BETWEEN cases.
Co-authored-by: Isaac
…ion divergence Add a comment at the new findCommonTypeForBinaryComparison nanos arms noting that AnyTimestampNanoType covers both LTZ and NTZ, so the nanos types honor castDatetimeToString like micros TimestampType (LTZ) -- deliberately more precise than micros TimestampNTZType, which has no arm and falls through to the config-blind canPromoteAsInBinaryComparison case. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add non-ANSI type-coercion arms so that a string compared against a nanosecond-precision timestamp column (
TIMESTAMP_NTZ(p)/TIMESTAMP_LTZ(p), p in [7, 9]) is cast to that nanos type, mirroring the existing microsecondTimestampTypehandling (StringPromotionTypeCoercionequality arms +TypeCoercion.findCommonTypeForBinaryComparison).Why are the changes needed?
Without a nanos arm, a string compared to a nanos column loses sub-microsecond precision under legacy
spark.sql.legacy.typeCoercion.datetimeToString.enabled=trueinstead of parsing at full precision like the microsecond types. ANSI mode is unchanged.Does this PR introduce any user-facing change?
Yes. String vs nanosecond-timestamp comparisons now coerce the string to the nanos type with full precision (preview
spark.sql.timestampNanosTypes.enabled).How was this patch tested?
Extended existing suits and added golden file tests.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code