Repository navigation
fix(charts): every chart runs where namespaces deny all traffic by default - #581
Conversation
…fault On a cluster whose namespaces start from a deny-all NetworkPolicy (ingress and egress, DNS included), most of the stack never came up: the policies were missing, off by default, switched off by the kind overlays, or did not select the pods that needed them. Tested on a Calico kind cluster with a Kyverno deny-all in every namespace: a full kates deploy, helm test on every release, a Litmus pod-delete (Pass) and a Velero backup location (Available). - monitoring 1.8.0: policies for the whole stack (DNS, API server, the namespace; open egress for Prometheus and Alertmanager; Grafana's UI) and kube-prometheus-stack's own operator and admission-Job policies. The operator and kube-state-metrics crash-looped without them. - velero 11.3.5: a policy (DNS, API server, object store out, metrics in), as a pre-install hook so the CRD Job has it; SeaweedFS's lab manifest gets one. - mirror-maker2 0.11.20: API server egress for the workers, which read their KafkaUser Secret through it and hung at start; a hook policy for the tests and the preflight and secret-sync Jobs. - kates-chaos 2.4.0: networkPolicy on by default, and a policy in each target namespace for the pods Litmus starts there (chaosUID), which hung at their first API call. - apicurio-registry 0.5.0, cdc-postgresql 0.3.0: networkPolicy on by default; the Helm tests can reach their pods. - kafka-cluster 1.0.27, kates 0.15.5, kafka-ui 0.3.3: the kind and generic overlays no longer switch policies off. kates gets a policy for its tests and cleanup Job; kafka-ui reaches Connect and the registry in any namespace. - headlamp 0.2.2, strimzi-operator 0.4.13: the Helm tests, and the strimzi CRD upgrade Job, get the egress they need. - connect-cluster 2.1.22: a hook policy for the preflight and secret-sync Jobs, through kafka-common 0.1.2's new kafka-common.netpol.jobs (with a test). - kates-platform 0.8.18 packs the new subcharts; the book's version table is regenerated. A release upgraded with --reuse-values keeps the old networkPolicy.enabled false of apicurio-registry, cdc-postgresql and kates-chaos; --reset-values picks up the new default. The velero and strimzi-operator hook policies stay behind on helm uninstall.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…e new defaults kafka-common.netpol.jobs emitted its Kafka rule even for an empty port list, which renders as a rule with no ports and no peers: all egress allowed. It is now left out, with a harness case. The render matrix expected the kind overlay to switch kafka-cluster's policies off (it keeps all five now) and read the watch-scope annotation off every strimzi-operator NetworkPolicy (it is on the operator's, not the new jobs policy).
…y kind DNS egress went to pods in any namespace, which drops a node-local DNS cache (GKE, AKS, EKS NodeLocal DNSCache): it answers from a node address that no pod selector matches. Every DNS rule, and kafka-common.netpol.dnsEgress without selectors, now allows port 53 at any address, as the Strimzi operator and Kates API policies already did. Headlamp named the ingress-nginx namespace in its template; it is networkPolicy.ingressNamespace now. Its API server egress went to 0.0.0.0/0, which an IPv6 API server never matches; it goes to any address on 443 and 6443 like the other charts.
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Code graph out of date: Health delta baseline: last indexed commit 0a08117, 7 commit(s) behind this PR's base. Impact and hotspot numbers below may be wrong.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 5 advisory finding(s) below merit a look before merge.
Not checked: tests were not run; no formal proof of the changed code.
Graphify review — findings
Turns networkPolicy.enabled on by default across the registry, CDC Postgres, kates-chaos and related charts, and adds the policies that let Helm tests and the preflight/secret-sync hook Jobs still work where a namespace denies all traffic. Apicurio and CDC Postgres now admit their own test pod, and the CDC Postgres default with empty allow lists admits 5432 from every namespace rather than just the test, so that default is where the broadest ingress gets decided. The new shared kafka-common.netpol.jobs template and the kates-chaos experiments policy open egress for Jobs and chaos pods, so what they select and what they allow when kafkaPorts or ipBlock is unset decides how much traffic they actually restrict.
Worth a look
- New target-namespace policy cuts chaos pods' egress where none was blocked before —
charts/kates-chaos/templates/networkpolicy.yaml:80· Escalate · high- NOT verified (no proof, no reproducing execution) — needs human review
- Enabled policy with empty allow lists now admits every namespace instead of only the Helm test —
charts/cdc-postgresql/templates/networkpolicy.yaml:21· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
- Empty kafkaPorts makes the hook-Job egress rule allow all traffic —
charts/kafka-common/templates/_netpol.tpl:155· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
- Jobs NetworkPolicy may select worker pods and widen their egress past perSource and ipBlock —
charts/mirror-maker2/templates/jobs-networkpolicy.yaml:11· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
- Kafka UI egress widened to every namespace —
charts/kafka-ui/templates/networkpolicy.yaml:84· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 951 functions depend on the 951 functions this change touches.
Health — grade A; 2 existing hotspot(s) in the area this change touches (pre-existing, not introduced here):
required— 0 callers, 10 callees (medium)required— 0 callers, 6 callees (medium)
Verification — 951 functions in the blast radius were not formally verified this run (proofs are advisory here).
Health delta baseline: last indexed commit 0a08117, 7 commit(s) behind this PR's base.
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 951 function(s) in the blast radius were not formally verified this run
…skips Strimzi pods The target-namespace policy narrowed every chaosUID pod to DNS and the API server, which in a namespace with no deny-all cut egress that was open before (probes against Kafka or HTTP endpoints). It now allows all egress, which is what lets them run where the namespace denies everything, and networkPolicy.targets.enabled=false turns it off (| default true read an explicit false as true). kafka-common.netpol.jobs never selects a pod with strimzi.io/kind, so a component label added through podLabels cannot widen the workers' egress past their own policy.
Why
On a cluster whose namespaces start from a deny-all NetworkPolicy (ingress and egress, DNS included), most of the stack never came up. The Prometheus operator and kube-state-metrics crash-looped, MirrorMaker 2 workers hung at start, Litmus runners hung at their first API call, Kafka's init container never reached the API server on kind, and most Helm tests failed. The policies were missing (monitoring, velero), off by default (apicurio-registry, cdc-postgresql, kates-chaos), switched off by the kind/generic overlays (kafka-cluster, kates, kates-chaos, kafka-ui), or did not select the pods that needed them (test pods, hook Jobs, Litmus engine pods).
What changes
networkPolicy.enabled(default true): DNS, API server and the namespace for every component; open egress for Prometheus and Alertmanager; Grafana UI and HTTPS; the Helm test.kube-prometheus-stack.prometheusOperator.networkPolicy.enabled: truecovers the operator webhook and the admission hook Jobsconfig/velero/seaweedfs.yamlgets its ownchaosUID Exists): DNS + API servervalues-kind.yamlno longer disables policiesvalues-kind.yamlno longer disables policies; policy for the tests and cleanup Job (noapplabel)connect); Helm test reaches the UI;values-generic.yamlno longer disables policieskafka-common.netpol.jobskafka-common.netpol.jobshelper, with a harness testThe book's version table (
docs/book/appendix-d-versions.md) is regenerated.How it was tested
A kind cluster from
config/cluster.yamlwith Calico instead of kindnet (kindnet's policy engine does not enforce on Docker Desktop: a deny-all pod still reached nginx), and a Kyverno ClusterPolicy generating a deny-all policy in every namespace except kube-system, kyverno and cert-manager.kates deploy --with-kafka-connect --with-mirror-maker2: every pod Ready.helm testpasses for kafka-cluster, strimzi-operator, monitoring, apicurio, kates-chaos, kafka-ui, mirror-maker2, cdc-postgresql, headlamp and velero.kafkacompletes with verdict Pass.chart_bumps.py, helm lint on every changed chart,check-chart-policies.sh(13/13 PASS), helm unittest for kafka-cluster, connect-cluster and the kafka-common harness,gen-version-matrix.sh --check, yamllint.Any Kubernetes cluster, not only kind
namespaceSelector: {}never matches. This covers every chart here,kafka-common.netpol.dnsEgresswithout selectors, the Kyverno-generatedallow-dns-egress, and the SeaweedFS and Litmus policies underconfig/.networkPolicy.ingressNamespacereplaces the hardcodedingress-nginx. Its API server egress goes to any address on 443/6443 instead of0.0.0.0/0, which misses an IPv6 API server.Not fixed here (not network policy)
test-apiexpects 200 from/q/openapiand gets 401.test-connectors: Debezium cannot create its publication in Postgres.helm upgrade.docker.io/bitnamilegacy/kubectl:1.34does not exist, so its CRD Job cannot pull; tested withupgradeCRDs=false.Not checked against main, so whether these are older is unconfirmed.
Upgrade notes
--reuse-valueskeeps the oldnetworkPolicy.enabled: falseof apicurio-registry, cdc-postgresql and kates-chaos;--reset-valuespicks up the new default.helm uninstall.