additional spark on k8s operator sensor for sparkapp log tail -f

[jdonnelly-apixio]

Description
I think it would be useful to have an additional sensor (or.. I'm not certain if it should be a sensor or something else) that would tail -f the spark app log. That log output in the airflow ui would make it a more user friendly way to see what the app is up to or why it failed.

The current poke sensor is here:
https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/sensors/spark_kubernetes.py

Use case / motivation

I was hoping to make it easy for developers to investigate spark app issues in the airflow ui.

Are you willing to submit a PR?

I plan on hacking something together that would do this at some point, but I will try to read up on docs and see if I can do it the correct way. If the correct way seams feasible, I will give it a shot.

Related Issues

n/a

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!