[SPARK-54223][PYTHON] Add task context and data metrics to Python runner logs

[Nishanth28]

What changes were proposed in this pull request?

Currently, the log messages in PythonRunner and related Python execution classes do not include Spark task context information during Python UDF execution.

This makes it harder to correlate Python worker timing metrics and data processing statistics with the specific Spark tasks that executed the UDFs, especially when debugging performance issues or data skew in production environments.

This improvement adds task context details along with data processing metrics to the log statements in PythonRunner and PythonUDFRunner classes to enhance traceability and debugging of Python UDF execution.

Current Behaviour

When examining executor logs, there is a disconnect between task execution logs and Python runner logs:

INFO PythonRunner: Times: total = 2529, boot = 478, init = 31, finish = 2020

❌ NO TASK CONTEXT - Cannot correlate with task 4188
❌ NO DATA METRICS - Cannot see records processed or data size

Expected Behaviour

After this enhancement, logs include task context information and data metrics:

INFO PythonRunner: Starting Python task execution (Stage 32, Attempt 0) - task 83.0 in stage 32 (TID 4188)

PythonRunner: Times: total = 2529, boot = 478, init = 31, finish = 2020 - Records: 10000, Data: 2.45 MB - task 83.0 in stage 32 (TID 4188)

✅ INCLUDES TASK CONTEXT - Easy to correlate with task 4188
✅ INCLUDES DATA METRICS - Shows 10000 records processed, 2.45 MB data

Why are the changes needed?

Enable seamless correlation between task execution and Python UDF operations:

Does this PR introduce any user-facing change?

No

How was this patch tested?

Run existing test suite:

./build/mvn -pl core -am test -DwildcardSuites=org.apache.spark.deploy.PythonRunnerSuite

Result:

Run completed in 1 second, 626 milliseconds.
Total number of tests run: 3
Suites: completed 2, aborted 0
Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 4.2.0-SNAPSHOT:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [  1.554 s]
[INFO] Spark Project Tags ................................. SUCCESS [  1.885 s]
[INFO] Spark Project Common Java Utils .................... SUCCESS [  2.938 s]
[INFO] Spark Project Common Utils ......................... SUCCESS [ 10.154 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  7.751 s]
[INFO] Spark Project Networking ........................... SUCCESS [01:02 min]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 11.541 s]
[INFO] Spark Project Variant .............................. SUCCESS [  2.006 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [  7.967 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  3.467 s]
[INFO] Spark Project Core ................................. SUCCESS [02:31 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:23 min
[INFO] Finished at: 2025-11-07T10:49:09+05:30
[INFO] ------------------------------------------------------------------------

Was this patch authored or co-authored using generative AI tooling?

No