Repository navigation
Conversation
The priority queue counts the node selector requirements a node matches, and matchFields requirements count as well as matchExpressions ones. The old name suggested only labels count.
The IP address tie-break compared the CIDR strings, so 10.0.0.0/16 sorted before 9.0.0.0/16, although the rule says the smaller IP address comes first. Compare the addresses instead.
After the match count, the allocatable Pod CIDRs and the node mask size, the allocator compared the node selector strings, then the IP addresses. The selector text is an arbitrary rule, and ClusterCIDRs with the same node selector tied on it, leaving the order to the heap. Compare the IP addresses, then the names. Names are unique, so the order is total.
occupyCIDRs skipped terminating ClusterCIDRs that select the node, which only makes sense for new allocations. A node using a ClusterCIDR that is being deleted then failed every sync, and after a restart lost its association with it, so the ClusterCIDR could be deleted while the node still used its range. With an overlapping ClusterCIDR ranking lower, the node got associated with that one instead. Include terminating ClusterCIDRs when occupying CIDRs a node already has, and rename the occupy parameter to forAllocation, as it only leaves out terminating ClusterCIDRs for new allocations.
The ClusterCIDRs without a node selector were appended to the matching ClusterCIDRs in the order the controller first saw them, and even when terminating, so a ClusterCIDR being deleted still got new allocations. A node carrying the internal default label also matched them as a selector, which listed each of them twice. Rank the catch-all ClusterCIDRs in their own priority queue, after the ClusterCIDRs that select the node, and leave the terminating ones out for new allocations, as for the other ClusterCIDRs.
The allocator keyed its CIDR map by the string of the label selector it converted each node selector into. ClusterCIDRs with the same selector string shared an entry, and deleting a ClusterCIDR had to parse its node selector again to find that entry. Key the map by ClusterCIDR name instead, and keep each node selector parsed in the entry. A ClusterCIDR without a node selector, or with one equal to the internal kubernetes.io/clusterCIDR=default selector, stays the catch-all one. Node selectors are matched as before.
occupyCIDRs occupied a node's Pod CIDRs one by one and moved on to the next ClusterCIDR at the first failure, without releasing the ones already occupied. A dual-stack ClusterCIDR whose IPv4 range contained the node's IPv4 Pod CIDR but whose IPv6 range did not kept the IPv4 CIDR allocated without having the node associated, so no release freed it until the controller restarted. Check every Pod CIDR against the ClusterCIDR's ranges first, with the condition MultiCIDRSet.Occupy applies, and skip the ClusterCIDR if one doesn't fit.
ReleaseCIDR released a node's Pod CIDRs from one ClusterCIDR, the first one in priority order among those whose node selector matched the node's current labels. A node synced after its labels changed, or after a ClusterCIDR ranking higher for it was created, can be associated with a ClusterCIDR that doesn't select it anymore, or with two ClusterCIDRs. The release then missed one of them: the Pod CIDRs leaked and the ClusterCIDR couldn't be deleted until the controller restarted. Release from every ClusterCIDR that has the node associated, whatever its node selector, rank or terminating state, and remove every association even if a release fails.
… with Every sync of a node with Pod CIDRs occupied them in the first ClusterCIDR that selects the node and can hold them, without checking whether the node was already associated with another one. After the node labels changed, or a ClusterCIDR ranking higher for the node was created, a sync associated the node with a second ClusterCIDR, which then counted the node and its CIDRs, or failed on every sync if no ClusterCIDR selecting the node could hold them. Occupy the Pod CIDRs again in a ClusterCIDR that has the node associated and whose ranges hold them, and only look for a candidate if there is none. The range check leaves out an association left over from a deleted node with the same name whose Pod CIDRs were elsewhere.
A ClusterCIDR whose node selector failed to parse was not loaded, and its sync was retried forever, although the node selector is immutable. Load such a ClusterCIDR with a node selector that selects no node, and report it with a Warning event on the ClusterCIDR. Events recorded before Run starts sending them are kept until then, so the ones of the ClusterCIDRs loaded at startup are not dropped.
The event recorder builds the reference to the object of an event from the client-go scheme, which doesn't know the ClusterCIDR types. ClusterCIDRs from the informer cache carry no kind, so the recorder dropped their events, e.g. the Warning event on a ClusterCIDR whose node selector is rejected, with "no kind is registered for the type v1.ClusterCIDR in scheme". Register the ClusterCIDR types in the client-go scheme.
The allocator converted every node selector into one label selector. That ANDed the terms instead of ORing them, matched matchFields against the node labels, and made an empty term select every node. Parse each node selector with k8s.io/component-helpers nodeaffinity: terms are ORed, requirements within a term are ANDed, matchFields match the node name, and an empty term matches no node. The match count is that of the largest matching term. A node selector that is nil or has no terms, as in nodeSelectorTerms: [], still selects every node, as the ClusterCIDR API documents, and makes the ClusterCIDR a catch-all one. The internal kubernetes.io/clusterCIDR=default selector is now an ordinary node selector.
… doesn't select the node occupyCIDRs only tried the ClusterCIDRs whose node selector selects the node, and the catch-all ones. A node whose Pod CIDRs came from a ClusterCIDR that doesn't select it anymore, after its labels changed or after an upgrade changed which nodes a node selector selects, had its CIDRs untracked: the ClusterCIDR could hand them to another node, and be deleted while the node used them. Node selectors decide where new CIDRs come from, not where existing ones are. If no ClusterCIDR that applies to the node can hold its Pod CIDRs, try the others, terminating ones included, in priority order, and report it with a Warning event on the node.
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mneverov The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind bug
What this PR does / why we need it:
Second part of the split of #177, as requested in #177 (comment). Stacked on #190, the bug fixes: only the last three commits belong here; review #190 first.
The allocator converts every ClusterCIDR node selector into one label selector. That ANDs the terms instead of ORing them, matches
matchFieldsagainst a label namedmetadata.name, and makes an empty term select every node (#36).32ea05e: each node selector is parsed withk8s.io/component-helpers/scheduling/corev1/nodeaffinity, one matcher per term:matchFieldsmatch the node name (metadata.name,In/NotInwith one value).v1API documents, a nil node selector and one without terms (nodeSelectorTerms: []) select every node: the ClusterCIDR is a catch-all one. The CRD rejectsnodeSelector: {}, sincenodeSelectorTermsis required.kubernetes.io/clusterCIDR=default, an internal selector of the in-tree controller, is an ordinary label requirement now.8294c39: the selector rules inexamples/README.md.2ebb15a: upgrade fallback. The new matching changes which nodes some existing ClusterCIDRs select. Without the fallback, a node whose ClusterCIDR no longer selects it loses the tracking of its Pod CIDRs at the upgrade restart, and the same CIDR can go to another node. Now, if no ClusterCIDR that applies to a node can hold its Pod CIDRs, the others are tried, terminating ones included, in priority order. Selectors decide where new CIDRs come from, not where existing ones are. A Warning eventClusterCIDRNotSelectingNodeon the node reports it. A ClusterCIDR whose node selector is rejected now (Fix ClusterCIDR occupy, release and ranking bugs #190 loads it selecting no node) keeps its nodes' Pod CIDRs this way, and isn't deleted while they use them.Behavior changes for existing ClusterCIDRs:
nodeSelectorTerms: []nodeSelectorTerms: [{}]matchFields metadata.name In [n1]metadata.nameabsent)n1matchFields metadata.name NotIn [n1]n1matchFieldswithExists,DoesNotExist,Gt,Lt, or not exactly one valuekubernetes.io/clusterCIDR In [default]In every row, nodes that already have Pod CIDRs keep them.
Verification
nodeSelectorTerms: []through the API server and checks that a node gets its Pod CIDRs from it. Every commit builds and passesgo vet, the tests and golangci-lint on its own (git rebase --exec), andmake manifests generateleaves no diff.2ebb15a(TestOccupyCIDRsInClusterCIDRNotSelectingNode,TestBootstrapClusterCIDRWithInvalidNodeSelector, the third case ofTestOccupyCIDRsPrefersClusterCIDRHoldingPodCIDRsWhole) fail with the fallback disabled.9983818) only in the empty-terms rule (code, tests, README), in the CRD doc comment, which keeps thev1text, ingo.mod/go.sumfrommain, and in code moved by Fix ClusterCIDR occupy, release and ranking bugs #190 (selectsNoNode, the position ofrecordEvent, comments).maincontroller from ClusterCIDRs withmatchFields metadata.name NotIn [node-1]andmatchFields DoesNotExiststay with their nodes, new nodes get free CIDRs, the ClusterCIDRs wait for their nodes on deletion, and Warning events are recorded. Without the fallback, new nodes got the existing nodes' CIDRs. Not rerun on this head; Add bats e2e tests for Pod CIDR allocation, release and the upgrade from v0.3.0 #185 runs it in CI once rebased.Not verified:
Which issue(s) this PR is related to:
Fixes #36
Related: #157, another fix for #36.
Special notes for your reviewer:
matchFieldskeys (onlymetadata.name) at admission isn't part of this PR. A CEL rule onnodeSelectorTermsexceeds the CEL cost budget. With another key,Inmatches no node andNotInmatches every node, as onmain.Does this PR introduce a user-facing change?
🤖 Generated with Claude Code