Skip to content

Fix ClusterCIDR occupy, release and ranking bugs - #190

Open
mneverov wants to merge 14 commits into
kubernetes-sigs:mainfrom
mneverov:fix/occupy-release-by-association
Open

mneverov wants to merge 14 commits into
kubernetes-sigs:mainfrom
mneverov:fix/occupy-release-by-association

Conversation

@mneverov

Copy link
Copy Markdown
Member

What type of PR is this?

/kind bug

What this PR does / why we need it:

Bug fixes split out of #177, as requested in #177 (comment). Node selectors are matched exactly as on main; the core v1 semantics (#36) follow in #177, stacked on this PR.

One commit per change, oldest first:

Map and matching (no behavior change)

  • fcc65f2: labelMatchCount → matchCount.
  • a03f18c: the CIDR map is keyed by ClusterCIDR name instead of by the label selector string, and each entry keeps its parsed node selector. ClusterCIDRs with the same selector string no longer share an entry, and deleting a ClusterCIDR no longer parses its node selector again. Matching stays main's conversion to a label selector, moved behind nodeSelector.match in node_selector.go. node_selector_test.go pins it, including where it differs from core v1. A nil selector and the internal kubernetes.io/clusterCIDR=default selector are still the catch-all.

Ranking

  • d8dd54b: the IP tie-break compares addresses, not strings. 9.0.0.0/16 ranks before 10.0.0.0/16.
  • 23046fc: ties are broken by IP address, then by name, instead of by the selector text. The order is total.
  • 980b282: catch-all ClusterCIDRs get their own priority queue, after the selecting ones. A terminating catch-all no longer gets new allocations.

Occupy and release

  • b07932f: a node keeps a terminating ClusterCIDR it uses, also across a restart. Before, every sync of such a node failed, and after a restart the ClusterCIDR could be deleted while the node used it.
  • 10e9eb2: a node's Pod CIDRs are occupied in a ClusterCIDR only if every one of them can be, checked before occupying any. Before, a dual-stack ClusterCIDR holding only the IPv4 CIDR kept it allocated without the node.
  • 71ec418: node deletion releases the Pod CIDRs from every ClusterCIDR the node is associated with, whatever its selector or rank. Before, after a relabel or a new higher-ranked ClusterCIDR, the CIDRs leaked and the ClusterCIDR couldn't be deleted until a restart.
  • 8fa9a99: a sync occupies a node's Pod CIDRs in the ClusterCIDR the node is already associated with, so it is never associated with a second one.
  • 703dfdd: a node's Pod CIDRs are occupied in a ClusterCIDR that holds them whole before one that only overlaps them: its range contains each of them, and its per-node blocks are no larger. Before, a ClusterCIDR with larger per-node blocks put the Pod CIDRs of two nodes in one block, released with either node, and a ClusterCIDR smaller than a Pod CIDR reserved only part of it. If none holds them whole, the overlapping one is still used, with a Warning event ClusterCIDRNotHoldingPodCIDRs on the node.
  • c179da2: at startup, a ClusterCIDR being deleted is loaded as terminating because of its deletion timestamp. Before, only its generation did that: the API server bumps it when it sets the deletion timestamp. A generation above 1 also comes from spec changes that pass the per-field immutability rules: ClusterCIDR spec fields can be added or removed despite the immutability rules #187.

Events

  • da6342a: a ClusterCIDR whose node selector is rejected, e.g. Gt with a non-integer value, is loaded selecting no node, with a Warning event InvalidNodeSelector, instead of being retried forever: the node selector is immutable. Events recorded before Run starts the event sink are kept until then, so the ones of the ClusterCIDRs loaded at startup are not dropped.
  • 7514c40: registers the ClusterCIDR types in the client-go scheme. Without it, the recorder dropped events on ClusterCIDRs from the informer cache. Found by the e2e tests in Add bats e2e tests for Pod CIDR allocation, release and the upgrade from v0.3.0 #185.
  • 05b12a1: stale comments and test names, and a duplicate CIDR set helper.

Verification

  • Unit and envtest tests. Every commit builds and passes go vet, the tests and golangci-lint on its own (git rebase --exec), and make manifests generate leaves no diff.
  • TestBootstrapClusterCIDRWithInvalidNodeSelector fails with the event buffering disabled.
  • No test relies on the new matching: the occupy and release tests use matchExpressions only, which match the same on main and in Match ClusterCIDR node selectors with the core v1 semantics #177.

Not verified on this branch, only on #177, which contains every commit here:

Known gaps, unchanged by this PR:

Which issue(s) this PR is related to:

None. The bugs reproduce on main and have no separate issues. #36 is fixed by #177.

Special notes for your reviewer:

The commits are meant to be reviewed one by one. a03f18c is the largest: it moves main's matching code into node_selector.go unchanged.

Does this PR introduce a user-facing change?

Fixed: Pod CIDRs leaked on node deletion after the node labels changed or a higher-ranked ClusterCIDR was created; a ClusterCIDR being deleted could be deleted while a node used it after a controller restart; a ClusterCIDR that could hold only some of a node's Pod CIDRs kept them allocated; a node's Pod CIDR could be reserved only in part, or share a per-node block with another node's Pod CIDR, when a ClusterCIDR with a smaller range or larger per-node blocks ranked first; a ClusterCIDR with an invalid node selector was retried forever, it is now loaded selecting no node and reported with a Warning event InvalidNodeSelector; events on ClusterCIDRs were dropped. Catch-all ClusterCIDRs are ranked by priority and a terminating one gets no new allocations. The IP address tie-break compares addresses, not strings.

🤖 Generated with Claude Code

The priority queue counts the node selector requirements a node
matches, and matchFields requirements count as well as matchExpressions
ones. The old name suggested only labels count.
The IP address tie-break compared the CIDR strings, so 10.0.0.0/16
sorted before 9.0.0.0/16, although the rule says the smaller IP address
comes first. Compare the addresses instead.
After the match count, the allocatable Pod CIDRs and the node mask size,
the allocator compared the node selector strings, then the IP addresses.
The selector text is an arbitrary rule, and ClusterCIDRs with the same
node selector tied on it, leaving the order to the heap.

Compare the IP addresses, then the names. Names are unique, so the
order is total.
occupyCIDRs skipped terminating ClusterCIDRs that select the node, which
only makes sense for new allocations. A node using a ClusterCIDR that is
being deleted then failed every sync, and after a restart lost its
association with it, so the ClusterCIDR could be deleted while the node
still used its range. With an overlapping ClusterCIDR ranking lower, the
node got associated with that one instead.

Include terminating ClusterCIDRs when occupying CIDRs a node already
has, and rename the occupy parameter to forAllocation, as it only
leaves out terminating ClusterCIDRs for new allocations.
The ClusterCIDRs without a node selector were appended to the matching
ClusterCIDRs in the order the controller first saw them, and even when
terminating, so a ClusterCIDR being deleted still got new allocations.
A node carrying the internal default label also matched them as a
selector, which listed each of them twice.

Rank the catch-all ClusterCIDRs in their own priority queue, after the
ClusterCIDRs that select the node, and leave the terminating ones out
for new allocations, as for the other ClusterCIDRs.
The allocator keyed its CIDR map by the string of the label selector it
converted each node selector into. ClusterCIDRs with the same selector
string shared an entry, and deleting a ClusterCIDR had to parse its node
selector again to find that entry.

Key the map by ClusterCIDR name instead, and keep each node selector
parsed in the entry. A ClusterCIDR without a node selector, or with one
equal to the internal kubernetes.io/clusterCIDR=default selector, stays
the catch-all one. Node selectors are matched as before.
occupyCIDRs occupied a node's Pod CIDRs one by one and moved on to the
next ClusterCIDR at the first failure, without releasing the ones
already occupied. A dual-stack ClusterCIDR whose IPv4 range contained
the node's IPv4 Pod CIDR but whose IPv6 range did not kept the IPv4 CIDR
allocated without having the node associated, so no release freed it
until the controller restarted.

Check every Pod CIDR against the ClusterCIDR's ranges first, with the
condition MultiCIDRSet.Occupy applies, and skip the ClusterCIDR if one
doesn't fit.
ReleaseCIDR released a node's Pod CIDRs from one ClusterCIDR, the first
one in priority order among those whose node selector matched the
node's current labels. A node synced after its labels changed, or after
a ClusterCIDR ranking higher for it was created, can be associated with
a ClusterCIDR that doesn't select it anymore, or with two ClusterCIDRs.
The release then missed one of them: the Pod CIDRs leaked and the
ClusterCIDR couldn't be deleted until the controller restarted.

Release from every ClusterCIDR that has the node associated, whatever
its node selector, rank or terminating state, and remove every
association even if a release fails.
… with

Every sync of a node with Pod CIDRs occupied them in the first ClusterCIDR
that selects the node and can hold them, without checking whether the
node was already associated with another one. After the node labels
changed, or a ClusterCIDR ranking higher for the node was created, a sync
associated the node with a second ClusterCIDR, which then counted the node
and its CIDRs, or failed on every sync if no ClusterCIDR selecting the
node could hold them.

Occupy the Pod CIDRs again in a ClusterCIDR that has the node associated
and whose ranges hold them, and only look for a candidate if there is
none. The range check leaves out an association left over from a deleted
node with the same name whose Pod CIDRs were elsewhere.
A ClusterCIDR whose node selector failed to parse was not loaded, and its
sync was retried forever, although the node selector is immutable.

Load such a ClusterCIDR with a node selector that selects no node, and
report it with a Warning event on the ClusterCIDR. Events recorded before
Run starts sending them are kept until then, so the ones of the
ClusterCIDRs loaded at startup are not dropped.
The event recorder builds the reference to the object of an event from
the client-go scheme, which doesn't know the ClusterCIDR types.
ClusterCIDRs from the informer cache carry no kind, so the recorder
dropped their events, e.g. the Warning event on a ClusterCIDR whose node
selector is rejected, with "no kind is registered for the type
v1.ClusterCIDR in scheme".

Register the ClusterCIDR types in the client-go scheme.
@kubernetes-prow kubernetes-prow Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/bug Categorizes issue or PR as related to a bug. labels Nov 18, 2026
@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mneverov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow kubernetes-prow Bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Nov 18, 2026
@kubernetes-prow kubernetes-prow Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant