You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a folder with a varying number of files I need to load into my database. Each file will need to go through an extract, transform, load (ETL) pipeline. So, I need to load each file one at a time though the ETL pipeline.
I have a dynamic task group that runs each group concurrently. But because the way we have structured the database, Airflow is triggering the same procedure multiple times, and the data is being overwritten. So, I need to run each group sequentially - with 1 file going through the 3 tasks before Airflow starts the next group.
But I cannot add max_active_tis_per_dag or max_active_tis_per_dagrun to the @task_group() or .partial() on the task group.
Would it be possible to add max_active_tis_per_dag or max_active_tis_per_dagrun to the task_group?
Or is there another way to achieve my goal?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
I have a folder with a varying number of files I need to load into my database. Each file will need to go through an extract, transform, load (ETL) pipeline. So, I need to load each file one at a time though the ETL pipeline.
I have a dynamic task group that runs each group concurrently. But because the way we have structured the database, Airflow is triggering the same procedure multiple times, and the data is being overwritten. So, I need to run each group sequentially - with 1 file going through the 3 tasks before Airflow starts the next group.
But I cannot add max_active_tis_per_dag or max_active_tis_per_dagrun to the
@task_group()
or.partial()
on the task group.Would it be possible to add max_active_tis_per_dag or max_active_tis_per_dagrun to the task_group?
Or is there another way to achieve my goal?
Example
Our Airflow version is currently 2.10.1 in some sort of docker environment.
This discussions 50297 may be similar.
Beta Was this translation helpful? Give feedback.
All reactions