Native DAG-Level Time Travel for Reproducible Backfills

[daniyarndh]

Currently, Airflow DAG execution logic is decoupled from its runtime execution state. When a DAG run fails or requires a historical backfill (e.g., re-running a task from 3 months ago), Airflow executes the task using today's live code base, current environment variables, and latest provider dependencies.

This leads to some significant operational challenges:

1. Unintended Side-Effects during Backfills: Code refactors or column renames deployed today will break or silently corrupt data when re-running historical DAG runs.
2. Reproducibility Friction: Debugging past task failures is difficult because the worker environment (Python package versions, DAG logic) has shifted since the failure occurred.
3. Auditability & Compliance: In regulated industries, proving the exact code version and transformation logic executed at a specific point in time requires manual correlation across Git history, CI logs, and Airflow task logs.

Proposed Solution

We want to introduce a Time-Travel Pipeline runs feature that takes a snapshot of the exact code and environment every time a pipeline starts. Airflow saves two unique identifiers directly to the run's metadata:

• The exact Git code commit (the logic)
• The exact container image ID (the software and dependencies)

In short, it turns pipeline re-runs into true time capsules, ensuring backfills are safe, bugs are reproducible, and past reports are fully auditable. Example:

{
  "execution_snapshot": {
    "git_commit": "4f82d1c67a",
    "docker_digest": "[registry.example.com/airflow-dags@sha256:a3f128c98e](https://registry.example.com/airflow-dags@sha256:a3f128c98e)...",
    "pinned_at": "2026-09-23T20:00:00Z"
  }
}

[potiuk]

That is a discussion that should happen on Devlist.