Airflog DAG Log-button not working with logs in s3 #27623
Replies: 4 comments 1 reply
-
|
Converted it into discussion as this is really a question not bug report. What log button ? On which tasks (current or past) ? Cand you please show screenshots and well - logs :) |
Beta Was this translation helpful? Give feedback.
-
|
After investigating further this seems to only happen when scheduler aborts a task for resource limiting reasons. It forgets to copy its log to s3 before aborting the task. See attachments for an example |
Beta Was this translation helpful? Give feedback.
-
|
After attaching the screenshot I also noticed that the time changed from 07:30 to 07:15 when clicking Log button. This might be related to the bug.
|
Beta Was this translation helpful? Give feedback.
-
|
And as you can see the logfile exists in s3, its just not being displayed by the "Log" button. |
Beta Was this translation helpful? Give feedback.
-
|
No. The problem it is a different log from different attempt. This is simply poor choice of integration. S3 has many limitations - it is not a file system, it is an object storage you cannot append stuf and cannot stream logs to it. If your task is terminated because of lack of resources, it cannot send the logs to S3 because it has no resources. So it's not "forgetting" about sending, it's just "impossibility" of sending the log or "impossibility" of recording information about the log file in S3 (sometime it might well be that s3 log was transmitted but information about it was not recorded). What you should do instead - if you have higher expectations that what you observe, you should use CloudWatch integration. It's more expensive - but for a good reason, because it handles streaming and in case task is interrupted - the log is already there, also logs are streamed directly from there (in case of s3 this is a hybrid solution where log is fetched from local worker during execution "live" but from S3 when the task is completed. This is not a bug, this is a natural limitation of S3 integration. |
Beta Was this translation helpful? Give feedback.
-
Apache Airflow version
2.4.2
What happened
I moved logs to s3 (set remote_log_conn_id = s3://my-bucket/logs & remote_logging = True) and it is working fine, when I click on the small squares in grid-view I can now view the logs from all of my connected airflow webservers (and I've verified that the logs actually are written to s3). So basically the "Logs" button is working fine.
But what is the "Log" button? It is all empty. Shouldn't this button show the same logs, just in full screen mode?
What you think should happen instead
I expected to see regular DAG logs
How to reproduce
Install airflow 2.4.2, configure log storage to S3 and click "Log" button after choosing one dag and viewing in grid view
Operating System
RedHat 8
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==6.0.0
apache-airflow-providers-common-sql==1.2.0
apache-airflow-providers-ftp==3.1.0
apache-airflow-providers-hashicorp==3.1.0
apache-airflow-providers-http==4.0.0
apache-airflow-providers-imap==3.0.0
apache-airflow-providers-odbc==3.1.2
apache-airflow-providers-oracle==3.4.0
apache-airflow-providers-slack==6.0.0
apache-airflow-providers-snowflake==3.3.0
apache-airflow-providers-sqlite==3.2.1
apache-airflow-providers-ssh==3.2.0
Deployment
Other
Deployment details
Installed in own pyenv using constraints-file
Anything else
always
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions