Skip to content

Add bats e2e tests for Pod CIDR allocation, release and the upgrade from v0.3.0 - #185

Draft
mneverov wants to merge 18 commits into
kubernetes-sigs:mainfrom
mneverov:test/e2e-bats
Draft

mneverov wants to merge 18 commits into
kubernetes-sigs:mainfrom
mneverov:test/e2e-bats

Conversation

@mneverov

@mneverov mneverov commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Stacked on #177: only the last two commits, b7852e7 and a189c15, belong to this PR (their diff). The PR targets main because #177's branch is in a fork. On hold until #177 merges; then rebase onto main.

The repository has no e2e tests of its own: e2e.yml runs the upstream Kubernetes networking conformance tests. The allocation and release bugs fixed in #177 only show with a real API server and controller: Pod CIDRs left allocated after node deletion, ClusterCIDRs deleted while nodes use them, and duplicate Pod CIDRs after an upgrade.

This PR adds bats tests in tests/, run by a new bats workflow on every pull request. kube-network-policies uses the same layout. The tests create fake Node objects (no kubelet) and ClusterCIDRs in a kind cluster.

tests/e2e_allocation.bats:

Test Bug on main
node deletion releases its Pod CIDR after a higher-ranked ClusterCIDR was created Pod CIDR leaked
node deletion releases its Pod CIDR after its labels changed release fails, ClusterCIDR undeletable
a relabeled node is not associated with a ClusterCIDR that selects it now second association
a node keeps a terminating ClusterCIDR rather than moving to an overlapping one node moves, overlapping ClusterCIDR leaks
a terminating ClusterCIDR is kept for its node across a controller restart ClusterCIDR deleted while in use
a relabeled node keeps its Pod CIDR across a controller restart duplicate Pod CIDR
a ClusterCIDR that can't hold all Pod CIDRs of a node keeps none of them IPv4 block stays allocated
a ClusterCIDR with an invalid node selector selects no node (main accepts the selector)
a node held by a finalizer keeps its Pod CIDR until it is gone duplicate Pod CIDR; skipped, still open on #177

tests/e2e_upgrade.bats installs v0.3.0 from oci://registry.k8s.io/networking/charts/node-ipam-controller, which converts node selectors to label selectors. Nodes get Pod CIDRs from a ClusterCIDR with matchFields metadata.name NotIn [node-1] and from one with matchFields DoesNotExist. The test then upgrades to the controller under test and checks:

  • No Pod CIDR is handed out twice.
  • Warning events ClusterCIDRNotSelectingNode (on the nodes) and InvalidNodeSelector (on the ClusterCIDR) are recorded.
  • Both ClusterCIDRs are deleted only after their nodes.

Delete this file once upgrades from v0.3.0 are no longer supported. From a release with the core v1 semantics, the setup can't happen.

Verification

Local kind cluster, bats 1.11.1:

Controller e2e_allocation.bats e2e_upgrade.bats
#177 (6262d6c) 8 pass, 1 skipped 3 pass
main (35eb93d) 8 fail, each at the assertion for its bug —
#177 without the upgrade fixes (48af180) — 3 fail: both duplicates, missing events, ClusterCIDR deleted while in use

The tests found a bug in #177: the InvalidNodeSelector event was dropped at runtime, because the client-go scheme didn't know the ClusterCIDR types. Fixed in #177 by 6262d6c.

On GitHub (run 42373305617, on a189c15), the bats e2e job passes in 7 minutes, cluster creation and image build included. It uses the kind, kubectl and helm preinstalled on ubuntu-latest. The first run failed while installing bats: bats-action resolved the latest release through the unauthenticated GitHub API and got an error page. a189c15 pins bats 1.11.1 and passes the workflow token.

Which issue(s) this PR is related to:

N/A. Covers the fixes of #177 (#36).

Special notes for your reviewer:

  • Run locally with bats -T tests/. CLUSTER_NAME reuses an existing cluster, e.g. the one of make run. See tests/README.md.
  • The helpers wait until the new controller pod holds the leader lease after an install or restart: a ready pod is not active yet.
  • Warning events are matched by object UID, so that events of an earlier object with the same name don't count.

Does this PR introduce a user-facing change?

NONE

🤖 Generated with Claude Code

The priority queue counts the node selector requirements a node
matches, and matchFields requirements count as well as matchExpressions
ones. The old name suggested only labels count.
The IP address tie-break compared the CIDR strings, so 10.0.0.0/16
sorted before 9.0.0.0/16, although the rule says the smaller IP address
comes first. Compare the addresses instead.
After the match count, the allocatable Pod CIDRs and the node mask size,
the allocator compared the node selector strings, then the IP addresses.
The selector text is an arbitrary rule, and ClusterCIDRs with the same
node selector tied on it, leaving the order to the heap.

Compare the IP addresses, then the names. Names are unique, so the
order is total.
occupyCIDRs skipped terminating ClusterCIDRs that select the node, which
only makes sense for new allocations. A node using a ClusterCIDR that is
being deleted then failed every sync, and after a restart lost its
association with it, so the ClusterCIDR could be deleted while the node
still used its range. With an overlapping ClusterCIDR ranking lower, the
node got associated with that one instead.

Include terminating ClusterCIDRs when occupying CIDRs a node already
has, and rename the occupy parameter to forAllocation, as it only
leaves out terminating ClusterCIDRs for new allocations.
The ClusterCIDRs without a node selector were appended to the matching
ClusterCIDRs in the order the controller first saw them, and even when
terminating, so a ClusterCIDR being deleted still got new allocations.
A node carrying the internal default label also matched them as a
selector, which listed each of them twice.

Rank the catch-all ClusterCIDRs in their own priority queue, after the
ClusterCIDRs that select the node, and leave the terminating ones out
for new allocations, as for the other ClusterCIDRs.
The allocator flattened every node selector into one label selector and
keyed its CIDR map by that string. That ANDed the terms instead of ORing
them, turned matchFields into label requirements, made empty selectors
select every node, and let ClusterCIDRs with the same selector string
share an entry.

Key the CIDR map by ClusterCIDR name instead, and keep each node
selector parsed with k8s.io/component-helpers nodeaffinity: terms are
ORed, requirements within a term are ANDed, matchFields match the node
name, and an empty term matches no node. The match count is that of the
largest matching term. A ClusterCIDR without a node selector is the
catch-all one, rather than one with the internal
kubernetes.io/clusterCIDR=default selector, which is now an ordinary
node selector.

An empty nodeSelector selects no node, as in the core v1 description of
nodeSelectorTerms in the same CRD. Correct its doc comment, which said
it selects every node.
occupyCIDRs occupied a node's Pod CIDRs one by one and moved on to the
next ClusterCIDR at the first failure, without releasing the ones
already occupied. A dual-stack ClusterCIDR whose IPv4 range contained
the node's IPv4 Pod CIDR but whose IPv6 range did not kept the IPv4 CIDR
allocated without having the node associated, so no release freed it
until the controller restarted.

Check every Pod CIDR against the ClusterCIDR's ranges first, with the
condition MultiCIDRSet.Occupy applies, and skip the ClusterCIDR if one
doesn't fit.
ReleaseCIDR released a node's Pod CIDRs from one ClusterCIDR, the first
one in priority order among those whose node selector matched the
node's current labels. A node synced after its labels changed, or after
a ClusterCIDR ranking higher for it was created, can be associated with
a ClusterCIDR that doesn't select it anymore, or with two ClusterCIDRs.
The release then missed one of them: the Pod CIDRs leaked and the
ClusterCIDR couldn't be deleted until the controller restarted.

Release from every ClusterCIDR that has the node associated, whatever
its node selector, rank or terminating state, and remove every
association even if a release fails.
… with

Every sync of a node with Pod CIDRs occupied them in the first ClusterCIDR
that selects the node and can hold them, without checking whether the
node was already associated with another one. After the node labels
changed, or a ClusterCIDR ranking higher for the node was created, a sync
associated the node with a second ClusterCIDR, which then counted the node
and its CIDRs, or failed on every sync if no ClusterCIDR selecting the
node could hold them.

Occupy the Pod CIDRs again in a ClusterCIDR that has the node associated
and whose ranges hold them, and only look for a candidate if there is
none. The range check leaves out an association left over from a deleted
node with the same name whose Pod CIDRs were elsewhere.
… doesn't select the node

occupyCIDRs only tried the ClusterCIDRs whose node selector selects the
node, and the catch-all ones. A node whose Pod CIDRs came from a
ClusterCIDR that doesn't select it anymore, after its labels changed or
after an upgrade changed which nodes a node selector selects, had its
CIDRs untracked: the ClusterCIDR could hand them to another node, and be
deleted while the node used them.

Node selectors decide where new CIDRs come from, not where existing ones
are. If no ClusterCIDR that applies to the node can hold its Pod CIDRs,
try the others, terminating ones included, in priority order, and report
it with a Warning event on the node. Events recorded before Run starts
sending them are kept until then, so the ones of the startup occupy are
not dropped.
A ClusterCIDR whose node selector failed to parse was not loaded, and its
sync was retried forever, although the node selector is immutable. Node
selectors accepted before they followed the core v1 semantics can be
rejected now while nodes use the ClusterCIDR's CIDRs: those CIDRs were
untracked and could be handed to other nodes, and deleting the
ClusterCIDR removed its finalizer at once.

Load such a ClusterCIDR with a node selector that selects no node, and
report it with a Warning event on the ClusterCIDR. It hands out no new
CIDRs, the CIDRs its nodes have are occupied in it, and it is not
deleted while nodes use them.
The event recorder builds the reference to the object of an event from
the client-go scheme, which doesn't know the ClusterCIDR types.
ClusterCIDRs from the informer cache carry no kind, so the recorder
dropped their events, e.g. the Warning event on a ClusterCIDR whose node
selector is rejected, with "no kind is registered for the type
v1.ClusterCIDR in scheme".

Register the ClusterCIDR types in the client-go scheme.
@kubernetes-prow kubernetes-prow Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. labels Nov 6, 2026
@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mneverov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow kubernetes-prow Bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 6, 2026
@mneverov

mneverov commented Nov 6, 2026

Copy link
Copy Markdown
Member Author

/hold

Stacked on #177, which has to merge first. This PR then gets rebased onto main.

@kubernetes-prow kubernetes-prow Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 6, 2026
…rom v0.3.0

The tests run against a kind cluster with fake Node objects and check
which Pod CIDRs the controller allocates and when it lets a ClusterCIDR
be deleted: after node labels changed, after a higher-ranked or
terminating ClusterCIDR, across a controller restart, for a ClusterCIDR
that can't hold all Pod CIDRs of a node, and for a node selector the
controller rejects. A node held by a finalizer is a known failure and
skipped.

The upgrade test installs v0.3.0, which converted node selectors to
label selectors, lets it allocate Pod CIDRs from ClusterCIDRs that
don't select the nodes under the core v1 semantics, upgrades to the
controller under test, and checks that no Pod CIDR is handed out twice.
Delete it once upgrades from v0.3.0 are no longer supported.

The bats workflow runs both on every pull request, following
kube-network-policies.
bats-action looked up the latest bats release through the unauthenticated GitHub API, which was rate limited on the runner: the download returned an error page and the job failed with "./install.sh: No such file or directory". Pin bats 1.11.1, pass the workflow token, and skip the bats libraries, which the tests don't use.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant