[Airflow 2.0] Using log-cleanup in Docker

[masbayuw]

Hi, i wanna ask, why it doesn't work for my airflow on docker?
The dags is ok but it won't delete my logs on local and so in the webserver it self

and this is modified log-cleanup.py

# # #32

[masbayuw]

the log shows this:

and in my docker the log folder is ready:

and I try to mount airflow.cfg into my docker, like this:

[alionar]

i think you need to set this Airflow Variable > 0: airflow_log_cleanup__max_log_age_in_days

https://github.com/teamclairvoyant/airflow-maintenance-dags/tree/master/log-cleanup#airflow-log-cleanuppy

airflow_log_cleanup__max_log_age_in_days - integer - Length to retain the log files if not already provided in the conf. If this is set to 30, the job will remove those files that are 30 days old or older.

[masbayuw]

i think you need to set this Airflow Variable > 0: airflow_log_cleanup__max_log_age_in_days

https://github.com/teamclairvoyant/airflow-maintenance-dags/tree/master/log-cleanup#airflow-log-cleanuppy

airflow_log_cleanup__max_log_age_in_days - integer - Length to retain the log files if not already provided in the conf. If this is set to 30, the job will remove those files that are 30 days old or older.

i set to 0 because i want to delete the log immediately. But i think i know my issue that base_log_folder that connect with airflow.cfg but i don't know how to mount that on my docker

[alionar]

what docker image you use for this airflow?

if you use official one, you can add this as env in your docker-compose (since you already mounted it as ./logs:/opt/airflow/logs)

AIRFLOW__LOGGING__BASE_LOG_FOLDER: '/opt/airflow/logs'

https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#base-log-folder

[masbayuw]

what docker image you use for this airflow?

if you use official one, you can add this as env in your docker-compose (since you already mounted it as ./logs:/opt/airflow/logs)

AIRFLOW__LOGGING__BASE_LOG_FOLDER: '/opt/airflow/logs'

https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#base-log-folder

I use this airflow 2.0.1
this is my docker images list

[masbayuw]

Thank you @alionar for your response, I'm glad with this community who help newbie just like me. I'm sorry if my english is not good enough to explain my problem. Ok now I'm gonna tell my problem is, I need a solution for my airflow-log cleanup and email opearator. I've try with another airflow os (airflow ova via oracle vm) that include airflow.cfg and it's work normaly. but when i try to run my dags with docker composer, there're no task done with my dags (the cleanup and email operator not work). I think this is due to the airflow cfg not working with docker-composer properly. I hope you don't mind with my question, and thank you for your explaination.

[miqbal23]

I'm also experiencing the same issue with log cleanup dag. I've already set the airflow_log_cleanup__max_log_age_in_days to 3 (days) and airflow_log_cleanup__enable_delete_child_log to True. But the dag didn't clean the logs on the airflow/logs/scheduler directory

And the logs showed there's no data inside the scheduler