Has anyone used javaagent on apache spark?

[imperio-wxm]

I am trying to connect opentelemetry to apache spark, but how the context and span created by driver transmit it to executors.

The idea is to broadcast traceid and spanid to each executor via broadcast variables, and then use propagation to extract traceid+spanid as context to continue using on the executor.

Is there a best practice for this?

[imperio-wxm]

Has anyone answered this question?

[trask]

hi @imperio-wxm, there is no auto-instrumentation for Apache Spark currently. you could try manual context propagation

[patpatpat123]

Hello team, two years after the original post, with Spark 4 and java agent 2, it is still not working.

Would it be possible to get some help on this initiative?

This would help each and every project avoid manual code to manage the context propagation.

Thank you