Conditional Task Skip option

[match-jasoncavnar]

Description

Add a keyword option at the task level to conditionally set if the task should be skipped on DAG run.

Use case/motivation

We would like to run our DAG in three environments; Dev, QA, and Prod. In Dev and QA we have sensors and file download tasks that would fail because the files do not exist in the environments. The skip keyword would allow us to conditionally set if the task should be skipped based on the DAG running in our DEV and QA envrionments.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[abhishekbhakat]

Why don't you just set different environment variables for your environments and use AirflowSkipException to skip a task.
Something like:

        if os.environ.get("ENVIRONMENT") == "dev":
            raise AirflowSkipException("Skipping this task in dev")

This doesn't need a feature.