[LIVY-1072] Reduce Kubernetes API load in SparkKubernetesApp polling - #545
[LIVY-1072] Reduce Kubernetes API load in SparkKubernetesApp polling#545soumyadeeplogin wants to merge 1 commit into
Conversation
soumyadeeplogin
changed the title
|
Filed https://issues.apache.org/jira/browse/LIVY-1072 for this change. |
|
@soumyadeeplogin Thank you for opening these PRs as your first contributions! Due to a few upgrades done in the last few days, CI tests can only run successfully when rebased to the latest commit. Could you please update your PRs to include the latest commits from the |
SparkKubernetesApp.getApplicationReport previously issued a single LIST returning every pod tagged with the session's app tag — on large jobs this deserializes hundreds of executor pods into memory each poll cycle, even though application state is derived only from the driver pod. Two changes: - Narrow the driver lookup to spark-role=driver at the API server so the LIST returns a single pod. - Gate the separate executor LIST on a new config flag livy.server.kubernetes.executor-tracking.enabled (default true, so existing deployments see no behavior change). Operators running large clusters can disable it to skip the executor LIST entirely. When disabled, per-executor Grafana/Loki log URLs and per-executor entries in session diagnostics are omitted; driver state tracking is unaffected. Adds two tests: default-enabled invariant and diagnostics graceful degradation when executors is empty. (cherry picked from commit 27ba941) Co-authored-by: Soumyadeep Roy <soumyadeep.roy@salesforce.com>
a8b8632 to
94b0237
Compare
|
Thanks @gyogal -- Just rebased and force-pushed all three PRs onto the latest master. They're clean/mergeable now. Since this is my first contribution, could you also approve the CI workflow when you get a chance? |
|
Thanks for the PR! Could you please add the required AI tooling disclosure to the PR description? Was this patch authored or co-authored using generative AI tooling?(If yes, include 'Generated-by: ' followed by the tool name and version. If no, write 'No'.) |
|
Thanks, @roczei! Added the AI tooling disclosure section to the PR description. |
5 participants
What changes were proposed in this pull request?
SparkKubernetesApp.getApplicationReportpreviously issued a single LIST returning every pod tagged with the session's app tag. On large jobs this deserializes hundreds of executor pods into memory on every poll cycle, even though application state is derived only from the driver pod.Two changes:
spark-role=driverat the API server so the LIST returns a single pod.livy.server.kubernetes.executor-tracking.enabled(defaulttrue, so existing deployments see no behavior change). Operators running large clusters can disable it to skip the executor LIST entirely. When disabled, per-executor Grafana/Loki log URLs and per-executor entries in session diagnostics are omitted; driver state tracking is unaffected.Adds two tests: default-enabled invariant, and diagnostics graceful degradation when the executor list is empty.
How was this patch tested?
Existing and newly added unit tests in
SparkKubernetesAppSpec, plus manual validation against a live Kubernetes cluster running Spark jobs with hundreds of executors.Was this patch authored or co-authored using generative AI tooling?
Yes. Generated-by: Claude Code (Sonnet 5, Anthropic), used to scan/plan the extraction of local UIP patches for upstreaming and to draft/iterate on the code and tests in this PR under human review. Please refer to the ASF Generative Tooling Guidance for details.
Note on JIRA
Filed as LIVY-1072.