FivetranOperator stuck in deferred state till it times out

[dgrewal-tc]

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.9.2

What happened?

This is in relation to astronomer/airflow-provider-fivetran-async#109

As mentioned in the comments, When we upgraded Airflow from 2.8.1 to 2.9.2, FivetranOperator in airflow-provider-fivetran-async package started giving issues abruptly.

We are using the FivetranOperator to trigger a Fivetran connector. The connector completes in 5-10 minutes, but the Airflow task does not finish. It remains in a deferred state until it times out and fails.

Airflow Version : 2.9.2
airflow-provider-fivetran-async==2.1.0

Failed logs

2024-09-15, 10:26:53 UTC] INFO - Pausing task as DEFERRED.
[2024-09-15, 10:26:54 UTC] INFO - Task exited with return code 100 (task deferral)

[2024-09-15, 11:26:58 UTC] INFO - Resuming after deferral
[2024-09-15, 11:26:59 UTC] INFO - Executing <Task(FivetranOperator)> on 2024-09-14 08:00:00+00:00
[2024-09-15, 11:26:59 UTC] INFO - Started process 9527 to run task
[2024-09-15, 11:26:59 UTC] INFO - Job 918648: Subtask
[2024-09-15, 11:26:59 UTC] ERROR - Task failed with exception
Traceback (most recent call last):
airflow.exceptions.AirflowTaskTimeout

Success logs

[2024-09-15, 12:12:51 UTC] INFO - Pausing task as DEFERRED. 
[2024-09-15, 12:12:52 UTC] INFO - Task exited with return code 100 (task deferral)

[2024-09-15, 12:12:52 UTC] INFO - Using connection ID 'id' for task execution.
[2024-09-15, 12:12:52 UTC] INFO - sync is still running...
[2024-09-15, 12:12:52 UTC] INFO - sleeping for 120 seconds.
[2024-09-15, 12:14:53 UTC] INFO - sync is still running...
[2024-09-15, 12:14:53 UTC] INFO - sleeping for 120 seconds.
[2024-09-15, 12:16:53 UTC] INFO - sync is still running...
[2024-09-15, 12:16:53 UTC] INFO - sleeping for 120 seconds.

[2024-09-15, 12:19:00 UTC] INFO - Resuming after deferral
[2024-09-15, 12:19:00 UTC] INFO - Executing <Task(FivetranOperator):> on 2024-09-14 08:00:00+00:00
[2024-09-15, 12:19:00 UTC] INFO - Started process 11601 to run task
[2024-09-15, 12:19:00 UTC] INFO - Job 918720: Subtask
[2024-09-15, 12:19:00 UTC] INFO - Fivetran connector finished syncing at 2024-09-15 12:17:07.840000+00:00
[2024-09-15, 12:19:00 UTC] INFO - Marking task as SUCCESS.
[2024-09-15, 12:19:00 UTC] INFO - Task exited with return code 0

Can someone please have a look ?

Thanks

What you think should happen instead?

No response

How to reproduce

Use FivetranOperator.
This issue does not happen every time we run it. It is very random.

Operating System

Amazon Linux. VERSION=2023

Versions of Apache Airflow Providers

No response

Deployment

Amazon (AWS) MWAA

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[eladkal]

Please elaborate why this is a bug in Airflow?
There were no similar reports on other providers thus my conclusion is that this is a bug in the specific operator that you are using. FivetranOperator is not official operator of Apache-Airflow.

We are not investigating 3rd party code. If you believe the issue is with Airflow itself please provide reproduce example on a service that we can check.

[potiuk]

Converting it to a discussion - this does not seem likely it's an Airflow issue, it's more of a 3rd-party - while we can still discuss it, this is not clear if it is an "airflow issue" - more evidence needs to be provided - if you can't do it @dgrewal-tc - reach out to whoever supports the provider you use so that they can investingate and provide more evidences.