feat(self-managed): isolate multiple control planes

[mikeyrcamp]

Customer Summary

Self-managed NVCF can run multiple isolated control planes in one Kubernetes
cluster by assigning each plane a stable identity and unique domain.

TL;DR

Self-managed NVCF deployments can run multiple independent control planes in
one Kubernetes cluster without namespace or resource collisions. A validated
control-plane identity scopes control-plane and compute-plane namespaces,
release names, routes, data-store state, operator watches, and lifecycle
operations. An empty identity preserves the existing single-control-plane
behavior.

Additional Details

• Derive plane-specific service namespaces, routes, Gateway references, OpenBao identities, Cassandra authorization state, and cluster registration profiles.
• Label managed resources with a stable control-plane identity and reject foreign ownership before reconcile, update, or deletion.
• Keep the shared NVCFBackend CRD in an explicit prerequisite release so removing one plane cannot remove the API used by another.
• Scope NVCA informers, requests, MiniServices, storage resources, workload namespaces, credentials, and cleanup to the owning plane.
• Replay matching source Secrets after informer synchronization so preexisting Secrets converge on cold start.
• Retry concurrent Secret update conflicts with a fresh ownership check, and propagate target failures instead of silently succeeding.
• Delete successful pre-delete hook RBAC so uninstall leaves no plane-specific cluster-scoped residue.
• Add unit, Helm rendering, migration, CLI, lifecycle, and local-cluster regression coverage.
• Wire the supported self-managed control-plane entrypoint so install, update,
  and teardown operate only on namespaces carrying the matching ownership
  label.
• Label named, stack-managed ClusterIssuers and remove only issuers owned by
  the selected plane. Require the canonical per-plane name for managed issuers
  while continuing to support shared external issuers.

Customer Release Notes

Self-managed NVCF supports multiple isolated control planes in one Kubernetes
cluster by assigning each plane a stable identity and unique domain.

Plan Summary

• Named mode derives resources from global.controlPlane.id.
• Legacy mode keeps existing names and behavior when the ID is empty.
• Shared cluster prerequisites remain explicit and independently managed.
• Destructive lifecycle commands verify ownership before uninstalling or
  deleting namespaces and retained managed issuers.

Usage

Configure a distinct identity and domain for each named control plane:

global:
  controlPlane:
    id: plane-a
  domain: plane-a.example.test

Install shared cert-manager, Gateway API, and observability prerequisites once,
then use the selected environment for every lifecycle command:

make install HELMFILE_ENV=plane-a
make apply HELMFILE_ENV=plane-a
make destroy HELMFILE_ENV=plane-a

Repeat with a different ID, domain, and environment for the second plane. See
the current installation guide and compute-plane README for the complete
command sequence and lifecycle safeguards.

For the Reviewer

Please focus on:

• legacy behavior when controlPlane.id is empty;
• ownership checks before reconcile and cleanup;
• intentional cluster-scoped resources versus accidental shared state;
• Secret informer startup replay and conflict retry semantics;
• OpenBao and Cassandra identity migration behavior;
• Gateway listener Certificate/Secret placement and per-plane PKI issuer/SAN
  derivation;
• source and vendored NVCA chart consistency.

For QA

Is QA needed? Yes.

Source checks run on the proposed PR tree:

(
  cd src/compute-plane-services/nvca
  go test \
    -ldflags '-X github.com/NVIDIA/k8s-dra-driver-gpu/internal/info.version=v25.8.0' \
    ./pkg/nvca/... ./cmd/nvca/... -count=1
)
bazel test //src/compute-plane-services/nvca/pkg/nvca/... //src/compute-plane-services/nvca/cmd/nvca/... --nocache_test_results
make -C deploy/helm/nvca-operator lint test-control-plane-isolation
deploy/helm/gateway-routes/scripts/check-llm-worker-routes.sh
deploy/helm/gateway-routes/scripts/test-render-routes.sh
helm lint deploy/helm/nvcf-pki
deploy/helm/nvcf-pki/scripts/check-render.sh
make -C deploy/stacks/self-managed test
make -C deploy/stacks/nvcf-compute-plane test-local
make -C tools/ncp-local-cluster test-isolated-control-plane-gateways
./tools/ci/check-docs
git diff --check origin/main

All checks passed except the full targeted Bazel selection on macOS. Twelve of
13 Bazel test targets passed. The remaining target failed only the unchanged
TestMetricsBackwardCompatibility check because the Darwin process collector
did not emit process_virtual_memory_bytes or
process_resident_memory_bytes. The direct Go suite and a focused Bazel rerun
of the changed control-plane and Secret-mirroring regressions passed. Linux CI
is pending.

An earlier reference implementation was exercised in a local Kubernetes
cluster with:

• two named control planes installed in separate namespaces;
• staged A-first and B-second upgrades while the other plane remained healthy and usable;
• preexisting Secret cold starts without a post-start source mutation;
• own-plane updates with byte-stable foreign-plane resources;
• direct and merged foreign ownership/namespace rejection;
• successful uninstall with no plane-specific namespaced or cluster-scoped residue.

Those results validate the design and regression scenarios. They are not
evidence that the final PR commit or its published artifacts have passed the
same live smoke test.

Notes

• This PR targets the current main branch only. It does not modify or
  backport a released-version branch or package.
• The self-managed manifest still pins the currently published
  helm-nvcf-pki chart. Named issuer live behavior requires publishing the
  chart change and updating the stack pin through the normal release process.
• This remains a draft pending coordinated publication of the updated
  first-party stack, charts, CLI, and images, followed by a non-production
  two-plane install, staged upgrade, and uninstall smoke test.

Issues

Closes #1481

Related Pull Requests

None yet.

Dependencies

No new or updated third-party dependencies. No NOTICE change is required.

Checklist

• I am familiar with the Contributing Guidelines.
• I have signed off my commits for Developer Certificate of Origin (DCO) compliance.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🌿 Preview your docs: https://nvidia-preview-fix-multi-control-plane-namespace-isolation.docs.buildwithfern.com/nvcf