run dag-processor as a job, only once

[antonio-antuan]

In our installation we don't need to run dag-processor periodically, it's enough to run it once right after deployment.
Currently dag-processor does not support in charts something like "-n 1". Actually it is supported but when process is finished - pod is restarted. I assume that the right way to do it is to run it as a job.

Is it possible to do it without any changes in airflow's helm chart?

[potiuk]

No. You need to run dag file processor continuously. This is required by airflow. One of the reasons is that in certain circumstances dag file processor also runs callbacks for tasks that have been forcefully killed - they are not only used to serialize DAGs - but also to preform this housekeeping while DAG to execute is parsed to memory as Python object.

Currently airflow does not support the configuration where dag file processor (or scheduler) periodic parsing is disabled. We do not plan to change it currently but maybe in the future we will change it. For now it's a basic assumption that dag file processor runs continuously. Disabling it will cause strange problems

[antonio-antuan]

I see, thanks.
What if I increase period of dags scanning? let's say, once an hour. Does it affect those callbacks in a way that they will be executed only once an hour when this "scanning" process run?

[potiuk]

Yes