You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This util dag is not deleting airflow logs for the master node but doing a delete for all worker nodes, please guide to do delete for the master node also where It is Airflow Cluster with Celery Executor. Thanks
The text was updated successfully, but these errors were encountered:
You can manually set up a cron job to delete log files in the master node. We have got a similar setup as yours. find /home/ubuntu/airflow/logs/*/* -type f -mtime +7 -exec rm -rf {} \;
We run this once every day.
@chetkhatri the DAG task deletes logs on the node on which it is running so that's the reason why only logs on worker nodes are being removed.
You can try the new airflow-log-cleanup-pwdless-ssh.py DAG and specify the hosts to run the process on Line 41.
Follow the readme for this DAG and make sure you're able to passwordless SSH onto the nodes from the worker nodes.
This util dag is not deleting airflow logs for the master node but doing a delete for all worker nodes, please guide to do delete for the master node also where It is Airflow Cluster with Celery Executor. Thanks
The text was updated successfully, but these errors were encountered: