Skip to content

Make RollupMapper calendar windows DST-aware#68150

Open
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:fix/rollup-dst-aware-windows-68004
Open

Make RollupMapper calendar windows DST-aware#68150
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:fix/rollup-dst-aware-windows-68004

Conversation

@Vamsi-klu
Copy link
Copy Markdown
Contributor

@Vamsi-klu Vamsi-klu commented Jun 7, 2026

What

Make DayWindow (the calendar window used by RollupMapper) DST-aware. With a UTC or naive upstream mapper a day is still exactly 24 hourly steps, so nothing changes. When paired with a local-timezone upstream mapper (e.g. StartOfDayMapper(timezone="America/New_York")) the window now enumerates the real local hours of the calendar day — 23 on a spring-forward day, 25 on a fall-back day — instead of a fixed 24.

The timezone is read from the upstream mapper (a tzinfo property added to the temporal mapper) and threaded into Window.to_upstream(..., tz) at call time, so windows stay stateless and serialization-free.

Why

DayWindow assumed every day has 24 hours. With a local-timezone upstream mapper, a spring-forward day has only 23 real hours, so one expected hourly key (e.g. 2024-03-10T02) never occurs — the required upstream key set could never be fully satisfied and the asset-partitioned rollup run was held forever. See #68004.

Impact

  • Local-timezone rollups across a spring-forward boundary are now satisfiable: the expected key set matches the hours that actually occur, so the run is no longer held indefinitely.
  • UTC / naive upstream mappers are unchanged (always 24 hourly steps), even on a DST-transition date.
  • No serialization change: the window carries no tz state; the timezone lives on the mapper (already serialized there), so a restored rollup reproduces the DST-correct set.
  • Scope / known limitation (fall-back day): the window enumerates 25 real local hours, but the repeated local 01:00 collapses to a single expected key unless the upstream mapper's input_format carries the UTC offset (%z) — so without %z it yields 24 distinct keys. The window no longer hangs; full 25-key coverage is left as follow-up work that also needs an encode_upstream change.

Testing

Reworked airflow-core/tests/unit/partition_mappers/test_window.py (TestDayWindow):

  • test_day_window_spring_forward_local_tz_yields_23_keys — replaces a former xfail test; asserts a spring-forward rollup (2024-03-10, America/New_York, %Y-%m-%dT%H) now expects exactly 23 upstream keys and that 2024-03-10T02 is absent.
  • test_day_window_fall_back_local_tz_does_not_hang — fall-back day (2024-11-03) yields 24 distinct keys (the repeated local 01:00 collapses to one key without %z); crucially none is unsatisfiable, so the rollup no longer hangs.
  • test_day_window_non_dst_local_day_yields_24_keys — a non-DST local day (2024-06-10) still yields 24 keys.
  • test_day_window_utc_yields_24_keys_on_dst_day — a UTC upstream mapper yields 24 keys even on the spring-forward date.
  • test_day_window_tz_none_yields_24_naive_steps (renamed) — with tz=None, DayWindow still yields a fixed 24 naive hourly steps on both DST dates.

Also updates the assets.rst docs and the example_asset_partition.py docstring to describe the DST-aware behavior, syncs the Task SDK DayWindow docstring (declaration only; the enumeration runs in airflow-core), and adds the 68150.bugfix.rst newsfragment.

closes: #68004


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

Vamsi-klu pushed a commit to Vamsi-klu/airflow that referenced this pull request Jun 7, 2026
The check-newsfragment-pr-number CI check requires the newsfragment file
to be named after the PR number, not the issue number.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The check-newsfragment-pr-number CI check requires the newsfragment file
to be named after the PR number, not the issue number.
@Vamsi-klu Vamsi-klu force-pushed the fix/rollup-dst-aware-windows-68004 branch from 2309bee to 98106c6 Compare June 7, 2026 01:05
The check-newsfragments-are-valid hook requires non-significant
newsfragments to be a single line.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make RollupMapper calendar windows DST-aware

1 participant