Graph "sub-view" that shows the immediate upstream and downstream dependencies of a given task

[notatallshaw-gts]

In large DAGs with lots of task instances that are highly coupled to each other it can sometimes be hard to "see" all the upstream and downstream of a given task instance. This can make it tricky to understand why a given task started so late or what tasks it caused to start late.

It would be nice if you could click on a task in the graph and choose something like "view immediate downstream and upstream tasks" and somewhere (either in-place or in a new tab) it would show you a "subgraph" containing the tasks of the immediate downstream and upstream tasks. You can then easily mouse over them to view their timings and see what the likely culprits are.

If you need to go further upstream or downstream you can apply this process again by select the task in the subgraph and again choosing "view immediate downstream and upstream tasks".

Anyway that's my pie in the sky idea from staring at a DAG for 20 minutes today until I understood what the problem was.

[potiuk]

I think this is mostly reflected in the new "dataset" views where you get exactly this in the newer version. Dataset is much more interesting as something to "focus" on rather than upstream/downstream task. But I will leave it to @bbovenzi and @pierrejeambrun to comment on other parts of the UI :)

[pierrejeambrun]

The filter upstream feature on the graph view seems to partially address what you are looking for. (~ for upstream tasks, we would still need to limit to the direct upstream tasks). Maybe we could complement this behavior to add downstream filtering capabilities as well as limiting downstream and upstream to directly connected tasks ?

[notatallshaw-gts]

Thanks! Somehow I missed when that button was added to Airflow!

[notatallshaw-gts]

It's not what I was describing, as "all upstream" tasks can still be very complicated (in fact it can be almost the whole DAG when I am looking at the one of the end tasks), but it would have made my job a little easier. Appreciate you pointing it out.

[notatallshaw-gts]

@potiuk @pierrejeambrun Having spent the last month+ heavily using "Filter Upstream" what I find missing is "Filter Downstream", as I usually want to check what tasks were impacted by a specific task, not just what tasks impacted a specific task.

I see an issue was also just created for this by someone else: #28847

[notatallshaw-gts]

A PR was created for this: #29226

Very excited, this covers all use cases I had for filtering on a complex DAG.

[notatallshaw-gts]

🥳 @yxiao1996 has extended the upstream filter to also do downstream which covers everything I think would be useful: #29226 (comment)