Skip to content

Match ClusterCIDR node selectors with the core v1 semantics - #191

Open
mneverov wants to merge 17 commits into
kubernetes-sigs:mainfrom
mneverov:fix/node-selector-v1-compat
Open

mneverov wants to merge 17 commits into
kubernetes-sigs:mainfrom
mneverov:fix/node-selector-v1-compat

Conversation

@mneverov

Copy link
Copy Markdown
Member

What type of PR is this?

/kind bug

What this PR does / why we need it:

Second part of the split of #177, as requested in #177 (comment). Stacked on #190, the bug fixes: only the last three commits belong here; review #190 first.

The allocator converts every ClusterCIDR node selector into one label selector. That ANDs the terms instead of ORing them, matches matchFields against a label named metadata.name, and makes an empty term select every node (#36).

  • 32ea05e: each node selector is parsed with k8s.io/component-helpers/scheduling/corev1/nodeaffinity, one matcher per term:
    • Terms are ORed; requirements within a term are ANDed.
    • matchFields match the node name (metadata.name, In/NotIn with one value).
    • An empty term selects no node.
    • The match count of a ClusterCIDR is that of its largest matching term.
    • As the v1 API documents, a nil node selector and one without terms (nodeSelectorTerms: []) select every node: the ClusterCIDR is a catch-all one. The CRD rejects nodeSelector: {}, since nodeSelectorTerms is required.
    • kubernetes.io/clusterCIDR=default, an internal selector of the in-tree controller, is an ordinary label requirement now.
  • 8294c39: the selector rules in examples/README.md.
  • 2ebb15a: upgrade fallback. The new matching changes which nodes some existing ClusterCIDRs select. Without the fallback, a node whose ClusterCIDR no longer selects it loses the tracking of its Pod CIDRs at the upgrade restart, and the same CIDR can go to another node. Now, if no ClusterCIDR that applies to a node can hold its Pod CIDRs, the others are tried, terminating ones included, in priority order. Selectors decide where new CIDRs come from, not where existing ones are. A Warning event ClusterCIDRNotSelectingNode on the node reports it. A ClusterCIDR whose node selector is rejected now (Fix ClusterCIDR occupy, release and ranking bugs #190 loads it selecting no node) keeps its nodes' Pod CIDRs this way, and isn't deleted while they use them.

Behavior changes for existing ClusterCIDRs:

Existing node selector Before After
nodeSelectorTerms: [] every node, ranked before the ClusterCIDRs without a node selector every node, ranked with them
only empty terms, e.g. nodeSelectorTerms: [{}] every node no node
several terms nodes matching all terms nodes matching any term
matchFields metadata.name In [n1] no node (label metadata.name absent) node n1
matchFields metadata.name NotIn [n1] every node every node but n1
matchFields with Exists, DoesNotExist, Gt, Lt, or not exactly one value every node or none rejected: selects no node, Warning event
kubernetes.io/clusterCIDR In [default] catch-all an ordinary label requirement

In every row, nodes that already have Pod CIDRs keep them.

Verification

  • Unit and envtest tests, including an envtest spec that creates a ClusterCIDR with nodeSelectorTerms: [] through the API server and checks that a node gets its Pod CIDRs from it. Every commit builds and passes go vet, the tests and golangci-lint on its own (git rebase --exec), and make manifests generate leaves no diff.
  • The tests of 2ebb15a (TestOccupyCIDRsInClusterCIDRNotSelectingNode, TestBootstrapClusterCIDRWithInvalidNodeSelector, the third case of TestOccupyCIDRsPrefersClusterCIDRHoldingPodCIDRsWhole) fail with the fallback disabled.
  • The tree differs from the head of Match ClusterCIDR node selectors with the core v1 semantics #177 (9983818) only in the empty-terms rule (code, tests, README), in the CRD doc comment, which keeps the v1 text, in go.mod/go.sum from main, and in code moved by Fix ClusterCIDR occupy, release and ranking bugs #190 (selectsNoNode, the position of recordEvent, comments).
  • The audit in the release note was checked with jq 1.8.1 against sample ClusterCIDRs covering each row of the table, plus a nil selector, a single-term selector, and a term next to an empty one, which it correctly leaves out.
  • Upgrade in kind, run on the head of Match ClusterCIDR node selectors with the core v1 semantics #177: Pod CIDRs allocated by the main controller from ClusterCIDRs with matchFields metadata.name NotIn [node-1] and matchFields DoesNotExist stay with their nodes, new nodes get free CIDRs, the ClusterCIDRs wait for their nodes on deletion, and Warning events are recorded. Without the fallback, new nodes got the existing nodes' CIDRs. Not rerun on this head; Add bats e2e tests for Pod CIDR allocation, release and the upgrade from v0.3.0 #185 runs it in CI once rebased.

Not verified:

  • Clusters with many ClusterCIDRs. The fallback scans all ClusterCIDRs once per node without an association, i.e. once per node per controller start. Not measured.

Which issue(s) this PR is related to:

Fixes #36

Related: #157, another fix for #36.

Special notes for your reviewer:

  • Validating matchFields keys (only metadata.name) at admission isn't part of this PR. A CEL rule on nodeSelectorTerms exceeds the CEL cost budget. With another key, In matches no node and NotIn matches every node, as on main.

Does this PR introduce a user-facing change?

action required: ClusterCIDR node selectors follow the core v1 NodeSelector semantics. Terms are ORed and the requirements within a term are ANDed; before, all requirements of all terms were ANDed. matchFields match the node name (metadata.name, In or NotIn with exactly one value); other matchFields are rejected, and such a ClusterCIDR selects no node (Warning event InvalidNodeSelector). An empty term selects no node. A ClusterCIDR without a node selector or with nodeSelectorTerms: [] is a catch-all one, ranked after the ClusterCIDRs that select the node. Nodes keep the Pod CIDRs they have, also in a ClusterCIDR that no longer selects them (Warning event ClusterCIDRNotSelectingNode on the node). Before upgrading, list the ClusterCIDRs whose matching changes:

kubectl get clustercidrs.networking.x-k8s.io -o json | jq -r '
  .items[]
  | select(.spec.nodeSelector != null)
  | .metadata.name as $name
  | (.spec.nodeSelector.nodeSelectorTerms // []) as $all
  | [$all[] | select(((.matchExpressions // []) + (.matchFields // [])) | length > 0)] as $terms
  | [$terms[] | .matchFields // [] | .[]] as $fields
  | if ($all | length) == 0 then "\($name): nodeSelectorTerms: [], used to rank before the ClusterCIDRs without a node selector, now ranks with them"
    elif ($terms | length) == 0 then "\($name): only empty terms, used to select every node, now selects none"
    elif any($fields[]; (.operator | IN("In", "NotIn") | not) or ((.values // []) | length) != 1) then "\($name): matchFields rejected now (only In/NotIn with one value), selects no node"
    elif any($terms[] | .matchExpressions // [] | .[]; .key == "kubernetes.io/clusterCIDR") then "\($name): uses kubernetes.io/clusterCIDR, no longer the catch-all marker"
    elif ($fields | length) > 0 then "\($name): uses matchFields, now matched against the node name instead of a label"
    elif ($terms | length) > 1 then "\($name): several terms, used to be ANDed, now ORed (selects more nodes)"
    else empty end'

🤖 Generated with Claude Code

The priority queue counts the node selector requirements a node
matches, and matchFields requirements count as well as matchExpressions
ones. The old name suggested only labels count.
The IP address tie-break compared the CIDR strings, so 10.0.0.0/16
sorted before 9.0.0.0/16, although the rule says the smaller IP address
comes first. Compare the addresses instead.
After the match count, the allocatable Pod CIDRs and the node mask size,
the allocator compared the node selector strings, then the IP addresses.
The selector text is an arbitrary rule, and ClusterCIDRs with the same
node selector tied on it, leaving the order to the heap.

Compare the IP addresses, then the names. Names are unique, so the
order is total.
occupyCIDRs skipped terminating ClusterCIDRs that select the node, which
only makes sense for new allocations. A node using a ClusterCIDR that is
being deleted then failed every sync, and after a restart lost its
association with it, so the ClusterCIDR could be deleted while the node
still used its range. With an overlapping ClusterCIDR ranking lower, the
node got associated with that one instead.

Include terminating ClusterCIDRs when occupying CIDRs a node already
has, and rename the occupy parameter to forAllocation, as it only
leaves out terminating ClusterCIDRs for new allocations.
The ClusterCIDRs without a node selector were appended to the matching
ClusterCIDRs in the order the controller first saw them, and even when
terminating, so a ClusterCIDR being deleted still got new allocations.
A node carrying the internal default label also matched them as a
selector, which listed each of them twice.

Rank the catch-all ClusterCIDRs in their own priority queue, after the
ClusterCIDRs that select the node, and leave the terminating ones out
for new allocations, as for the other ClusterCIDRs.
The allocator keyed its CIDR map by the string of the label selector it
converted each node selector into. ClusterCIDRs with the same selector
string shared an entry, and deleting a ClusterCIDR had to parse its node
selector again to find that entry.

Key the map by ClusterCIDR name instead, and keep each node selector
parsed in the entry. A ClusterCIDR without a node selector, or with one
equal to the internal kubernetes.io/clusterCIDR=default selector, stays
the catch-all one. Node selectors are matched as before.
occupyCIDRs occupied a node's Pod CIDRs one by one and moved on to the
next ClusterCIDR at the first failure, without releasing the ones
already occupied. A dual-stack ClusterCIDR whose IPv4 range contained
the node's IPv4 Pod CIDR but whose IPv6 range did not kept the IPv4 CIDR
allocated without having the node associated, so no release freed it
until the controller restarted.

Check every Pod CIDR against the ClusterCIDR's ranges first, with the
condition MultiCIDRSet.Occupy applies, and skip the ClusterCIDR if one
doesn't fit.
ReleaseCIDR released a node's Pod CIDRs from one ClusterCIDR, the first
one in priority order among those whose node selector matched the
node's current labels. A node synced after its labels changed, or after
a ClusterCIDR ranking higher for it was created, can be associated with
a ClusterCIDR that doesn't select it anymore, or with two ClusterCIDRs.
The release then missed one of them: the Pod CIDRs leaked and the
ClusterCIDR couldn't be deleted until the controller restarted.

Release from every ClusterCIDR that has the node associated, whatever
its node selector, rank or terminating state, and remove every
association even if a release fails.
… with

Every sync of a node with Pod CIDRs occupied them in the first ClusterCIDR
that selects the node and can hold them, without checking whether the
node was already associated with another one. After the node labels
changed, or a ClusterCIDR ranking higher for the node was created, a sync
associated the node with a second ClusterCIDR, which then counted the node
and its CIDRs, or failed on every sync if no ClusterCIDR selecting the
node could hold them.

Occupy the Pod CIDRs again in a ClusterCIDR that has the node associated
and whose ranges hold them, and only look for a candidate if there is
none. The range check leaves out an association left over from a deleted
node with the same name whose Pod CIDRs were elsewhere.
A ClusterCIDR whose node selector failed to parse was not loaded, and its
sync was retried forever, although the node selector is immutable.

Load such a ClusterCIDR with a node selector that selects no node, and
report it with a Warning event on the ClusterCIDR. Events recorded before
Run starts sending them are kept until then, so the ones of the
ClusterCIDRs loaded at startup are not dropped.
The event recorder builds the reference to the object of an event from
the client-go scheme, which doesn't know the ClusterCIDR types.
ClusterCIDRs from the informer cache carry no kind, so the recorder
dropped their events, e.g. the Warning event on a ClusterCIDR whose node
selector is rejected, with "no kind is registered for the type
v1.ClusterCIDR in scheme".

Register the ClusterCIDR types in the client-go scheme.
The allocator converted every node selector into one label selector. That
ANDed the terms instead of ORing them, matched matchFields against the node
labels, and made an empty term select every node.

Parse each node selector with k8s.io/component-helpers nodeaffinity: terms
are ORed, requirements within a term are ANDed, matchFields match the node
name, and an empty term matches no node. The match count is that of the
largest matching term.

A node selector that is nil or has no terms, as in nodeSelectorTerms: [],
still selects every node, as the ClusterCIDR API documents, and makes the
ClusterCIDR a catch-all one. The internal kubernetes.io/clusterCIDR=default
selector is now an ordinary node selector.
… doesn't select the node

occupyCIDRs only tried the ClusterCIDRs whose node selector selects the
node, and the catch-all ones. A node whose Pod CIDRs came from a
ClusterCIDR that doesn't select it anymore, after its labels changed or
after an upgrade changed which nodes a node selector selects, had its
CIDRs untracked: the ClusterCIDR could hand them to another node, and be
deleted while the node used them.

Node selectors decide where new CIDRs come from, not where existing ones
are. If no ClusterCIDR that applies to the node can hold its Pod CIDRs,
try the others, terminating ones included, in priority order, and report
it with a Warning event on the node.
@kubernetes-prow kubernetes-prow Bot added release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. kind/bug Categorizes issue or PR as related to a bug. labels Nov 18, 2026
@kubernetes-prow
kubernetes-prow Bot requested review from aojea and thockin November 18, 2026 14:55
@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mneverov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. release-note-action-required Denotes a PR that introduces potentially breaking changes that require user action. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NodeSelectorAsSelector needs to be fixed

1 participant