Dependent DAG Based schedules

[phaneendrareddy345]

Description

It would be really great, If we can schedule DAG's based on Upstream Dags.

For example, dag_c is dependent on dag_a and dag_b. Instead of using sensors and waiting for these 2 upstream tasks to finish, what if we can simply mention schedule is dependent on dag_a, dag_b something similar to outlets

I have implement something similar outside of airflow using GCP BigQuery, Cloud Function and Pub Sub. This is to schedule dags based on table events, that means whenever a specific table gets loaded the respective dependent dag will be triggered, I have created a variable in .py file called bq_table_dependecies to parse the dag dependencies.

Similar way to we do this internally with Airflow dag, that would be really helpful, the same feature is available in legacy scheduler like Control M

You can watch my youtube video here: https://youtu.be/nLdTAp9n5Gg
Blog: https://phaneendrareddyp.medium.com/airflow-custom-event-based-bigquery-table-dependencies-09597727ae35

Please reach out to [email protected] for further discussions and clarifications

Use case/motivation

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[coolbeans201]

Isn't that what data-aware scheduling accomplishes? https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/datasets.html

[potiuk]

Yep. Converting it into discussion because dataset aware scheduling is there and also a number of improvements to it are planned in Airflow 3. https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-73+Expanded+Data+Awareness

[phaneendrareddy345]

Apologies looks like the data aware scheduling solves the problem, thanks