Dynamic dags vs static dags for multiple clients

[gudata]

We have 500+ dags. Each dag is handling one client. All dags are the same, they have different "client-id" variable. Clients are in a database.

Now we generate the dags with a "generated_dags" dag. The generate dags gets the list of all clients and with some template it produces static dags. This works very well and we do not have any problems so far.

Recently we have the idea to dynamically generate the dags and having only one DAG which will generate sub-dags on the fly.

As a drawbacks:

• I know dags should be fast and there should be not database calls in it. what will happen if airflow needs to compile the dags and the previous dag is not finished?
• Now I have files, dags records and everything is visible, clear and easy to maintain. With the dynamic dags those will be dynamic and I think debugging and logging will be nightmare. I won't see dags files, there will be no source code etc.

Cons:

• dags will be updated every 30 sec, by AIRFLOW__SCHEDULER__MIN_FILE_PROCESS_INTERVAL=30, AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL=300

What is your experience on that?

[gudata]

Does anyone has experience working with dynamic and static dags?

I found one more drawback - the history might be lost if using dynamic dags.