Repository navigation
Conversation
The priority queue counts the node selector requirements a node matches, and matchFields requirements count as well as matchExpressions ones. The old name suggested only labels count.
The IP address tie-break compared the CIDR strings, so 10.0.0.0/16 sorted before 9.0.0.0/16, although the rule says the smaller IP address comes first. Compare the addresses instead.
After the match count, the allocatable Pod CIDRs and the node mask size, the allocator compared the node selector strings, then the IP addresses. The selector text is an arbitrary rule, and ClusterCIDRs with the same node selector tied on it, leaving the order to the heap. Compare the IP addresses, then the names. Names are unique, so the order is total.
occupyCIDRs skipped terminating ClusterCIDRs that select the node, which only makes sense for new allocations. A node using a ClusterCIDR that is being deleted then failed every sync, and after a restart lost its association with it, so the ClusterCIDR could be deleted while the node still used its range. With an overlapping ClusterCIDR ranking lower, the node got associated with that one instead. Include terminating ClusterCIDRs when occupying CIDRs a node already has, and rename the occupy parameter to forAllocation, as it only leaves out terminating ClusterCIDRs for new allocations.
The ClusterCIDRs without a node selector were appended to the matching ClusterCIDRs in the order the controller first saw them, and even when terminating, so a ClusterCIDR being deleted still got new allocations. A node carrying the internal default label also matched them as a selector, which listed each of them twice. Rank the catch-all ClusterCIDRs in their own priority queue, after the ClusterCIDRs that select the node, and leave the terminating ones out for new allocations, as for the other ClusterCIDRs.
The allocator flattened every node selector into one label selector and keyed its CIDR map by that string. That ANDed the terms instead of ORing them, turned matchFields into label requirements, made empty selectors select every node, and let ClusterCIDRs with the same selector string share an entry. Key the CIDR map by ClusterCIDR name instead, and keep each node selector parsed with k8s.io/component-helpers nodeaffinity: terms are ORed, requirements within a term are ANDed, matchFields match the node name, and an empty term matches no node. The match count is that of the largest matching term. A ClusterCIDR without a node selector is the catch-all one, rather than one with the internal kubernetes.io/clusterCIDR=default selector, which is now an ordinary node selector. An empty nodeSelector selects no node, as in the core v1 description of nodeSelectorTerms in the same CRD. Correct its doc comment, which said it selects every node.
occupyCIDRs occupied a node's Pod CIDRs one by one and moved on to the next ClusterCIDR at the first failure, without releasing the ones already occupied. A dual-stack ClusterCIDR whose IPv4 range contained the node's IPv4 Pod CIDR but whose IPv6 range did not kept the IPv4 CIDR allocated without having the node associated, so no release freed it until the controller restarted. Check every Pod CIDR against the ClusterCIDR's ranges first, with the condition MultiCIDRSet.Occupy applies, and skip the ClusterCIDR if one doesn't fit.
ReleaseCIDR released a node's Pod CIDRs from one ClusterCIDR, the first one in priority order among those whose node selector matched the node's current labels. A node synced after its labels changed, or after a ClusterCIDR ranking higher for it was created, can be associated with a ClusterCIDR that doesn't select it anymore, or with two ClusterCIDRs. The release then missed one of them: the Pod CIDRs leaked and the ClusterCIDR couldn't be deleted until the controller restarted. Release from every ClusterCIDR that has the node associated, whatever its node selector, rank or terminating state, and remove every association even if a release fails.
… with Every sync of a node with Pod CIDRs occupied them in the first ClusterCIDR that selects the node and can hold them, without checking whether the node was already associated with another one. After the node labels changed, or a ClusterCIDR ranking higher for the node was created, a sync associated the node with a second ClusterCIDR, which then counted the node and its CIDRs, or failed on every sync if no ClusterCIDR selecting the node could hold them. Occupy the Pod CIDRs again in a ClusterCIDR that has the node associated and whose ranges hold them, and only look for a candidate if there is none. The range check leaves out an association left over from a deleted node with the same name whose Pod CIDRs were elsewhere.
… doesn't select the node occupyCIDRs only tried the ClusterCIDRs whose node selector selects the node, and the catch-all ones. A node whose Pod CIDRs came from a ClusterCIDR that doesn't select it anymore, after its labels changed or after an upgrade changed which nodes a node selector selects, had its CIDRs untracked: the ClusterCIDR could hand them to another node, and be deleted while the node used them. Node selectors decide where new CIDRs come from, not where existing ones are. If no ClusterCIDR that applies to the node can hold its Pod CIDRs, try the others, terminating ones included, in priority order, and report it with a Warning event on the node. Events recorded before Run starts sending them are kept until then, so the ones of the startup occupy are not dropped.
A ClusterCIDR whose node selector failed to parse was not loaded, and its sync was retried forever, although the node selector is immutable. Node selectors accepted before they followed the core v1 semantics can be rejected now while nodes use the ClusterCIDR's CIDRs: those CIDRs were untracked and could be handed to other nodes, and deleting the ClusterCIDR removed its finalizer at once. Load such a ClusterCIDR with a node selector that selects no node, and report it with a Warning event on the ClusterCIDR. It hands out no new CIDRs, the CIDRs its nodes have are occupied in it, and it is not deleted while nodes use them.
The event recorder builds the reference to the object of an event from the client-go scheme, which doesn't know the ClusterCIDR types. ClusterCIDRs from the informer cache carry no kind, so the recorder dropped their events, e.g. the Warning event on a ClusterCIDR whose node selector is rejected, with "no kind is registered for the type v1.ClusterCIDR in scheme". Register the ClusterCIDR types in the client-go scheme.
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mneverov The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Member
Author
|
/hold Stacked on #177, which has to merge first. This PR then gets rebased onto |
mneverov
force-pushed
the
test/e2e-bats
branch
from
November 7, 2026 06:50
a0abb11 to
f53985e
Compare
mneverov
force-pushed
the
test/e2e-bats
branch
from
November 7, 2026 12:37
f53985e to
a189c15
Compare
…rom v0.3.0 The tests run against a kind cluster with fake Node objects and check which Pod CIDRs the controller allocates and when it lets a ClusterCIDR be deleted: after node labels changed, after a higher-ranked or terminating ClusterCIDR, across a controller restart, for a ClusterCIDR that can't hold all Pod CIDRs of a node, and for a node selector the controller rejects. A node held by a finalizer is a known failure and skipped. The upgrade test installs v0.3.0, which converted node selectors to label selectors, lets it allocate Pod CIDRs from ClusterCIDRs that don't select the nodes under the core v1 semantics, upgrades to the controller under test, and checks that no Pod CIDR is handed out twice. Delete it once upgrades from v0.3.0 are no longer supported. The bats workflow runs both on every pull request, following kube-network-policies.
bats-action looked up the latest bats release through the unauthenticated GitHub API, which was rate limited on the runner: the download returned an error page and the job failed with "./install.sh: No such file or directory". Pin bats 1.11.1, pass the workflow token, and skip the bats libraries, which the tests don't use.
mneverov
force-pushed
the
test/e2e-bats
branch
from
November 10, 2026 14:41
a189c15 to
fc9dba4
Compare
This was referenced Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind cleanup
What this PR does / why we need it:
Stacked on #177: only the last two commits,
b7852e7anda189c15, belong to this PR (their diff). The PR targetsmainbecause #177's branch is in a fork. On hold until #177 merges; then rebase ontomain.The repository has no e2e tests of its own:
e2e.ymlruns the upstream Kubernetes networking conformance tests. The allocation and release bugs fixed in #177 only show with a real API server and controller: Pod CIDRs left allocated after node deletion, ClusterCIDRs deleted while nodes use them, and duplicate Pod CIDRs after an upgrade.This PR adds bats tests in
tests/, run by a newbatsworkflow on every pull request. kube-network-policies uses the same layout. The tests create fake Node objects (no kubelet) and ClusterCIDRs in a kind cluster.tests/e2e_allocation.bats:mainmainaccepts the selector)tests/e2e_upgrade.batsinstalls v0.3.0 fromoci://registry.k8s.io/networking/charts/node-ipam-controller, which converts node selectors to label selectors. Nodes get Pod CIDRs from a ClusterCIDR withmatchFields metadata.name NotIn [node-1]and from one withmatchFields DoesNotExist. The test then upgrades to the controller under test and checks:ClusterCIDRNotSelectingNode(on the nodes) andInvalidNodeSelector(on the ClusterCIDR) are recorded.Delete this file once upgrades from v0.3.0 are no longer supported. From a release with the core v1 semantics, the setup can't happen.
Verification
Local kind cluster, bats 1.11.1:
e2e_allocation.batse2e_upgrade.bats6262d6c)main(35eb93d)48af180)The tests found a bug in #177: the
InvalidNodeSelectorevent was dropped at runtime, because the client-go scheme didn't know the ClusterCIDR types. Fixed in #177 by6262d6c.On GitHub (run 42373305617, on
a189c15), thebats e2ejob passes in 7 minutes, cluster creation and image build included. It uses the kind, kubectl and helm preinstalled onubuntu-latest. The first run failed while installing bats:bats-actionresolved the latest release through the unauthenticated GitHub API and got an error page.a189c15pins bats 1.11.1 and passes the workflow token.Which issue(s) this PR is related to:
N/A. Covers the fixes of #177 (#36).
Special notes for your reviewer:
bats -T tests/.CLUSTER_NAMEreuses an existing cluster, e.g. the one ofmake run. Seetests/README.md.Does this PR introduce a user-facing change?
🤖 Generated with Claude Code