ExternalTask: Check if the most recent task succeeds

[gabrielwry]

Description

A sensor that only relies on the most recent run to be in allowed_states instead of using a execution_delta or execution_delta_fn.

I can think of couple potential issues with this:

• we may need to sort the task_instance tables based on execution_date, can be expensive
• a race condition, where when our sensor is poking the external task is not in allowed_states, and when sensor retries, the previous task that we are supposed to check is in a allowed_states, but there is a new task being scheduled and is not in allowed_states again.

Any thoughts?

Use case/motivation

A sensor that only relies on the most recent run to be in allowed_states instead of using a execution_delta or execution_delta_fn.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[uranusjr]

Needs to define “recent”. What time should be used to compare runs—logical time, trigger time, finish time, or otherwise? Does this include manually triggered runs? If so, how should the times between manual and scheduled runs be compared?

[sfc-gh-syeager]

@uranusjr

First, estimate the usecase for the most recent run, as opposed to ExternalTaskSensor, which looks for a run at a specific logical time. The usecase in question isn't worried about historical runs, meaning that the most recent contains all the data and references desired.

Second, compare OPs request to the current version of the ExternalTaskSensor class, and the usecase presented.

What time should be used to compare runs?

The logical time. That's how the current class does it.

Does it include manually triggered runs? If so, how should the times between manual and scheduled runs be compared?

It doesn't look like the current class differentiates, so this one probably shouldn't either. Remember the most recent run (logical time-wise) contains all the references needed. If the logical time on a manual run is later than the logical time on any scheduled run, then the manual run is likely the one desired.