[SPARK-57821][SQL] Support nanosecond-precision timestamps in date_trunc#57224
[SPARK-57821][SQL] Support nanosecond-precision timestamps in date_trunc#57224stevomitric wants to merge 2 commits into
Conversation
Extend date_trunc (TruncTimestamp) to accept the nanosecond-precision timestamp types TIMESTAMP_NTZ(p)/TIMESTAMP_LTZ(p) (p in [7,9]) in addition to the microsecond TimestampType. The result now carries the input's nanosecond type/family, and truncation floors epochMicros via the existing truncTimestamp (NTZ evaluated at UTC, LTZ in the session zone) while always zeroing the sub-microsecond remainder, since date_trunc's finest unit is MICROSECOND. A new DateTimeUtils.truncTimestampNanos delegates to truncTimestamp so behavior stays identical to the micro path on the shared range. Why: the nanosecond timestamp preview (SPARK-56822) needs date_trunc to work end-to-end without silently downcasting to microseconds. Co-authored-by: Isaac
date_trunc's @ExpressionDescription had no nanosecond example, unlike its nanos-extended peers in this file (DayOfYear, Year, Quarter, Month, DayOfMonth, DayOfWeek, NanosToTimestamp, UnixNanos), so `DESCRIBE FUNCTION EXTENDED date_trunc` did not surface nanosecond support. Add a matching example gated on spark.sql.timestampNanosTypes.enabled that truncates a TIMESTAMP_NTZ nanos literal to MICROSECOND (dropping the sub-microsecond remainder). Extend the scalastyle directive to line.contains.tab, since the SET-output line carries a literal tab, matching the peer examples. Co-authored-by: Isaac
stevomitric
changed the title
|
cc @MaxGekk, @cloud-fan PTAL. |
cloud-fan
left a comment
cloud-fan
left a comment
There was a problem hiding this comment.
0 blocking, 0 non-blocking, 0 nits.
Clean, well-scoped extension of date_trunc to nanosecond timestamps, following the established nanos-expression pattern (TimeAdd/TimestampAddInterval/SecondWithFractionNanos): dataType-based branching in a single expression class, zoneIdForType for zone selection, and a delegating truncTimestampNanos helper in DateTimeUtils. LGTM.
Verification
Traced the semantics across the type families: nanos input keeps its exact type/family (NTZ->NTZ, LTZ->LTZ, precision preserved); micro TIMESTAMP_NTZ is deliberately left out of the accepted type set so the analyzer still casts it to micro LTZ (matching the inline comment), and zoneIdForType sends the NTZ family to UTC and everything else to the session zone. truncTimestampNanos reuses the existing truncTimestamp for the epoch-micros floor (inheriting all DST/offset/level handling) and zeroes nanosWithinMicro since MICROSECOND is date_trunc's finest unit. The LTZ golden output correctly shifts to the session zone, including the DAY-crosses-local-day case, and eval/codegen parity is exercised for both families. All hold.
2 participants
What changes were proposed in this pull request?
Extend
date_trunc(TruncTimestamp) to accept nanosecond-precision timestamps (TIMESTAMP_NTZ(p)/TIMESTAMP_LTZ(p), p in [7, 9]) in addition to microsecondTimestampType. The result keeps the input's nanos type/family; truncation floorsepochMicros(NTZ at UTC, LTZ in the session zone) and always zeroes the sub-microsecond remainder, sincedate_trunc's finest unit is MICROSECOND. AddsDateTimeUtils.truncTimestampNanosdelegating to the existingtruncTimestamp.Why are the changes needed?
date_truncpreviously narrowed a nanosecond argument to microseconds, dropping the sub-microsecond digits and forcing an LTZ result even for NTZ input.Does this PR introduce any user-facing change?
Yes.
date_trunc(fmt, ts_nanos)now returns the same nanosecond type/family with correct truncation (previewspark.sql.timestampNanosTypes.enabled).How was this patch tested?
DateExpressionsSuiteandtimestamp-{ltz,ntz}-nanos.sqlgolden files.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code