[WIP][SPARK-57816][SQL] Support nanosecond-precision timestamps in date_format, to_char and to_varchar#57225
Open
stevomitric wants to merge 1 commit into
Open
Conversation
…rmat, to_char and to_varchar
Extend DateFormatClass (which backs date_format, to_char and to_varchar) to
accept AnyTimestampNanoType inputs. Both interpreted eval and doGenCode render
via the TimestampNanosVal object carrier: TIMESTAMP_NTZ(p) renders its wall
clock zone-independently while TIMESTAMP_LTZ(p) renders in the formatter's
session zone, and fractional-second digits below the type precision floor to
zeros rather than being dropped. Also guard SimplifyDateTimeConversions from
rewriting a nanosecond-timestamp child, since the round trip through micro
TimestampType would silently truncate sub-microsecond precision.
Adds unit coverage in DateExpressionsSuite and SimplifyDateTimeConversionsSuite,
end-to-end SQL/Column coverage in TimestampNanosFunctionsSuiteBase, and golden
cases in timestamp-{ltz,ntz}-nanos.sql.
Co-authored-by: Isaac
stevomitric
force-pushed
the
stevomitric/spark-57816-nanos
branch
from
daa97ad to
bd14624
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Extend
DateFormatClass(which backsdate_format,to_charandto_varchar) to accept nanosecond-precision timestamps (TIMESTAMP_NTZ(p)/TIMESTAMP_LTZ(p), p in [7, 9]). Both eval and codegen render via the nanosecond-awareTimestampFormatter: NTZ renders its wall clock zone-independently, LTZ renders in the session zone, and sub-precision digits floor to zeros. Also guardsSimplifyDateTimeConversionsfrom rewriting a nanosecond child, whose round-trip through microsecondTimestampTypewould silently truncate.Why are the changes needed?
Nanosecond input was narrowed to microseconds, so the sub-microsecond digits never reached the formatter.
Does this PR introduce any user-facing change?
Yes.
date_format/to_char/to_varcharnow render nanosecond timestamps at full precision (previewspark.sql.timestampNanosTypes.enabled).How was this patch tested?
DateExpressionsSuite,SimplifyDateTimeConversionsSuite,TimestampNanosFunctionsSuiteBase, andtimestamp-{ltz,ntz}-nanos.sqlgolden files.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code