Upstreaming and pull request strategy for Spark on Kubernetes

[erikerlandson]

We'll be wanting a plan for presenting this project in PRs that are as digestible as possible by the Apache Spark reviewers upstream. We can use this issue to formulate a strategy

[erikerlandson]

The RSS seems like one good candidate for a PR, to be presented "after" the kube-spark core. If so, then possibly RSS-related config might require temporary disabling for initial core PR.

[erikerlandson]

The hadoop repo seems like another.

[erikerlandson]

Shuffle service / D.A. ?

[foxish]

The candidate set of different and separable components, and a potential order in which we could upstream them is:

1. Test infrastructure (minikube on jenkins)
2. Scheduler Backend + Submission Client
3. Dynamic allocation
4. Resource staging server
5. Kerberos support

(2) is the most complex of the lot. It mostly makes sense together, but we could break it down further if we absolutely must.

[foxish]

Following up on the discussion today in the meeting, the questions we need to answer are:

Test infrastructure

• Discovery - What's the right place to run our integration tests? AMPLab Jenkins? Externally? - @ssuchter
• Current resource requirements & running tests in parallel. - @mccheah
• Can we run tests on an external k8s cluster (is this okay from the upstream perspective?) @ssuchter/@foxish to follow up.

Tag 2.2 branch

• Merge fixes
• Cut a bug-fix release and announce. (@kimoonkim)

Scheduler Backend + Submission Client

• Steps
  • Submission orchestrator
  • Submission steps
  • Scheduler Backend
  • (@mcheah to assess if this makes sense for now, or if we more of a split)

cc @felixcheung

[mccheah]

It's also good to document the changes that we want to include in upstream that we haven't merged into branch-2.2-kubernetes yet, including:

• Resilience to SPARK-21642 - Conflict with upstream SPARK-21642 #482, PR Use a headless service to give a hostname to the driver. #483
• Move shuffle directory mounting for executors to a separate class - PR Extract more of the shuffle management to a different class. #454
• Unit tests for the scheduler backend - PR Unit Tests for KubernetesClusterSchedulerBackend #459

Any other ones I'm missing?

[foxish]

@mccheah all of those have merged I think.

For the 2.3 code-freeze, this seems to me like a reasonable (maybe slightly ambitious) target -
I expect we can speed up after the first PR.

• PR 1: Basic scheduler backend PR (Spark on Kubernetes - basic scheduler backend #498, [SPARK-18278] [Scheduler] Spark on Kubernetes - Basic Scheduler Backend apache/spark#19468)
  • Unit testing for executorpodfactory (Add unit-testing for executorpodfactory #491)
• PR 2: Basic submission client PR - covers spark-submit changes and Client.scala + the minimum set of submission steps and orchestrator ([SPARK-22646] [Submission] Spark on Kubernetes - basic submission client apache/spark#19717, Spark on Kubernetes - basic submission client #545)
  • Improve Unit testing for SparkSubmit changes
  • Unit tests for Client.scala
  • Reference Dockerfiles for driver and executor.
• PR 2.5: Docker images
  • ASF procedure for pre-built images
• PR 3: Init-container setup + make-distribution setup
  • Improve Unit testing (TBD)
• PR 4: Dynamic allocation PR
  • Improve Unit testing (TBD)
• PR 6: Documentation
• PR ? Parallel effort on Integration Tests
  • Discovery - RiseLAB CI
  • Will the ASF allow us to run on external GKE/GCE cluster?

Thoughts? If we all agree, I'll post this on the JIRA.