Airflow unable to handle more than 16 tasks at once

[atusman24]

Hi,

We have Airflow version 2.5.0 and we're trying to execute a job that contains 60 tasks which can be executed in parallel. Despite trying several concurrency variable values in Airlfow service and environment settings, the airflow is still unable to execute 60 tasks in parallel.
While executing, it starts out with 32 tasks(screenshot 1,2,3).

And after a few moments 16 out of the 32 change status to 'up_for_try' state(screenshot 4,5) and eventually get queued again.

On the other hand these 32 tasks that are initiated at once are still being executed in ECS as shown in screenshot 6. So it seems like those tasks are still executing while Aiflow is unable to keep track of them and sets them to be rerun.

Any help would be appreciated. Thank you

[potiuk]

A lot depends on resources you have and your deployment.

You have not explained which executor you have, how many machines you have, what is the parallelism for those machines, what kind of tasks you have them. Airflow as you see - is correclty scheduling the tasks, now it's your job as deployment manager, to make sure that you have enough workers (for celery) with enough resources (CPU/machines, memory, sometimes GPUS etc) to be able to run as many parallel processes you migh want.

In Kubernetes - you need to have enough resources to be able to run as many PODs you have.

Depending on the capacity of your system, the number of parallel things that might be executed. It's also your job to make sure that you allocate appropriately the resources (for example via K8S so I suggest you look at your deployment and make sure those resources are not blocking you.

[potiuk]

I can't understand either - your explanation does not make it clear. You stil have not explained how many workers and how much concurrency you have, and it is impossible to get into your head as a deployment manager managing your environment and help you.

[rraghunathan-exelixis]

@atusman24 are you able to find out why it is queued state? I'm facing the similar issue, im running 40 tasks per dag, but it is only running 23-25 tasks per run, all others are struck in queued state. There is not proper logs on any of the components.

these are my values,
celery:
worker_concurrency: 50
core:
parallelism: 128
max_active_tasks_per_dag: 50
max_active_runs_per_dag: 50

Please let me know

[Peter4net]

hey, @rraghunathan-exelixis did you figured it out? similar issue here

[gudata]

We had the same problem on airflow 2.6 or 2.9 (don't remember)

With a dedicated pool with 256 tasks. I start 50 ec2 machines with enough ram assigned to that pool and only 10 active tasks.

The problem gone when we upgraded to the new airflow 3 without changing anything.

The setup is: celery with airflow in containers from the example compose.yaml

[rraghunathan-exelixis]

hey, @rraghunathan-exelixis did you figured it out? similar issue here

Hey @Peter4net No luck so far. I'm still not able to figure out the resolution. I'm okay with the tasks being in a queued state, but after some time, once the workers have finished their previous execution, they should pick them up — which is not happening. Any luck on your end?