Repository navigation
Conversation
The priority queue counts the node selector requirements a node matches, and matchFields requirements count as well as matchExpressions ones. The old name suggested only labels count.
The IP address tie-break compared the CIDR strings, so 10.0.0.0/16 sorted before 9.0.0.0/16, although the rule says the smaller IP address comes first. Compare the addresses instead.
After the match count, the allocatable Pod CIDRs and the node mask size, the allocator compared the node selector strings, then the IP addresses. The selector text is an arbitrary rule, and ClusterCIDRs with the same node selector tied on it, leaving the order to the heap. Compare the IP addresses, then the names. Names are unique, so the order is total.
occupyCIDRs skipped terminating ClusterCIDRs that select the node, which only makes sense for new allocations. A node using a ClusterCIDR that is being deleted then failed every sync, and after a restart lost its association with it, so the ClusterCIDR could be deleted while the node still used its range. With an overlapping ClusterCIDR ranking lower, the node got associated with that one instead. Include terminating ClusterCIDRs when occupying CIDRs a node already has, and rename the occupy parameter to forAllocation, as it only leaves out terminating ClusterCIDRs for new allocations.
The ClusterCIDRs without a node selector were appended to the matching ClusterCIDRs in the order the controller first saw them, and even when terminating, so a ClusterCIDR being deleted still got new allocations. A node carrying the internal default label also matched them as a selector, which listed each of them twice. Rank the catch-all ClusterCIDRs in their own priority queue, after the ClusterCIDRs that select the node, and leave the terminating ones out for new allocations, as for the other ClusterCIDRs.
The allocator keyed its CIDR map by the string of the label selector it converted each node selector into. ClusterCIDRs with the same selector string shared an entry, and deleting a ClusterCIDR had to parse its node selector again to find that entry. Key the map by ClusterCIDR name instead, and keep each node selector parsed in the entry. A ClusterCIDR without a node selector, or with one equal to the internal kubernetes.io/clusterCIDR=default selector, stays the catch-all one. Node selectors are matched as before.
occupyCIDRs occupied a node's Pod CIDRs one by one and moved on to the next ClusterCIDR at the first failure, without releasing the ones already occupied. A dual-stack ClusterCIDR whose IPv4 range contained the node's IPv4 Pod CIDR but whose IPv6 range did not kept the IPv4 CIDR allocated without having the node associated, so no release freed it until the controller restarted. Check every Pod CIDR against the ClusterCIDR's ranges first, with the condition MultiCIDRSet.Occupy applies, and skip the ClusterCIDR if one doesn't fit.
ReleaseCIDR released a node's Pod CIDRs from one ClusterCIDR, the first one in priority order among those whose node selector matched the node's current labels. A node synced after its labels changed, or after a ClusterCIDR ranking higher for it was created, can be associated with a ClusterCIDR that doesn't select it anymore, or with two ClusterCIDRs. The release then missed one of them: the Pod CIDRs leaked and the ClusterCIDR couldn't be deleted until the controller restarted. Release from every ClusterCIDR that has the node associated, whatever its node selector, rank or terminating state, and remove every association even if a release fails.
… with Every sync of a node with Pod CIDRs occupied them in the first ClusterCIDR that selects the node and can hold them, without checking whether the node was already associated with another one. After the node labels changed, or a ClusterCIDR ranking higher for the node was created, a sync associated the node with a second ClusterCIDR, which then counted the node and its CIDRs, or failed on every sync if no ClusterCIDR selecting the node could hold them. Occupy the Pod CIDRs again in a ClusterCIDR that has the node associated and whose ranges hold them, and only look for a candidate if there is none. The range check leaves out an association left over from a deleted node with the same name whose Pod CIDRs were elsewhere.
A ClusterCIDR whose node selector failed to parse was not loaded, and its sync was retried forever, although the node selector is immutable. Load such a ClusterCIDR with a node selector that selects no node, and report it with a Warning event on the ClusterCIDR. Events recorded before Run starts sending them are kept until then, so the ones of the ClusterCIDRs loaded at startup are not dropped.
The event recorder builds the reference to the object of an event from the client-go scheme, which doesn't know the ClusterCIDR types. ClusterCIDRs from the informer cache carry no kind, so the recorder dropped their events, e.g. the Warning event on a ClusterCIDR whose node selector is rejected, with "no kind is registered for the type v1.ClusterCIDR in scheme". Register the ClusterCIDR types in the client-go scheme.
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mneverov The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind bug
What this PR does / why we need it:
Bug fixes split out of #177, as requested in #177 (comment). Node selectors are matched exactly as on
main; the core v1 semantics (#36) follow in #177, stacked on this PR.One commit per change, oldest first:
Map and matching (no behavior change)
fcc65f2:labelMatchCount→matchCount.a03f18c: the CIDR map is keyed by ClusterCIDR name instead of by the label selector string, and each entry keeps its parsed node selector. ClusterCIDRs with the same selector string no longer share an entry, and deleting a ClusterCIDR no longer parses its node selector again. Matching staysmain's conversion to a label selector, moved behindnodeSelector.matchinnode_selector.go.node_selector_test.gopins it, including where it differs from core v1. A nil selector and the internalkubernetes.io/clusterCIDR=defaultselector are still the catch-all.Ranking
d8dd54b: the IP tie-break compares addresses, not strings.9.0.0.0/16ranks before10.0.0.0/16.23046fc: ties are broken by IP address, then by name, instead of by the selector text. The order is total.980b282: catch-all ClusterCIDRs get their own priority queue, after the selecting ones. A terminating catch-all no longer gets new allocations.Occupy and release
b07932f: a node keeps a terminating ClusterCIDR it uses, also across a restart. Before, every sync of such a node failed, and after a restart the ClusterCIDR could be deleted while the node used it.10e9eb2: a node's Pod CIDRs are occupied in a ClusterCIDR only if every one of them can be, checked before occupying any. Before, a dual-stack ClusterCIDR holding only the IPv4 CIDR kept it allocated without the node.71ec418: node deletion releases the Pod CIDRs from every ClusterCIDR the node is associated with, whatever its selector or rank. Before, after a relabel or a new higher-ranked ClusterCIDR, the CIDRs leaked and the ClusterCIDR couldn't be deleted until a restart.8fa9a99: a sync occupies a node's Pod CIDRs in the ClusterCIDR the node is already associated with, so it is never associated with a second one.703dfdd: a node's Pod CIDRs are occupied in a ClusterCIDR that holds them whole before one that only overlaps them: its range contains each of them, and its per-node blocks are no larger. Before, a ClusterCIDR with larger per-node blocks put the Pod CIDRs of two nodes in one block, released with either node, and a ClusterCIDR smaller than a Pod CIDR reserved only part of it. If none holds them whole, the overlapping one is still used, with a Warning eventClusterCIDRNotHoldingPodCIDRson the node.c179da2: at startup, a ClusterCIDR being deleted is loaded as terminating because of its deletion timestamp. Before, only its generation did that: the API server bumps it when it sets the deletion timestamp. A generation above 1 also comes from spec changes that pass the per-field immutability rules: ClusterCIDR spec fields can be added or removed despite the immutability rules #187.Events
da6342a: a ClusterCIDR whose node selector is rejected, e.g.Gtwith a non-integer value, is loaded selecting no node, with a Warning eventInvalidNodeSelector, instead of being retried forever: the node selector is immutable. Events recorded beforeRunstarts the event sink are kept until then, so the ones of the ClusterCIDRs loaded at startup are not dropped.7514c40: registers the ClusterCIDR types in the client-go scheme. Without it, the recorder dropped events on ClusterCIDRs from the informer cache. Found by the e2e tests in Add bats e2e tests for Pod CIDR allocation, release and the upgrade from v0.3.0 #185.05b12a1: stale comments and test names, and a duplicate CIDR set helper.Verification
go vet, the tests and golangci-lint on its own (git rebase --exec), andmake manifests generateleaves no diff.TestBootstrapClusterCIDRWithInvalidNodeSelectorfails with the event buffering disabled.matchExpressionsonly, which match the same onmainand in Match ClusterCIDR node selectors with the core v1 semantics #177.Not verified on this branch, only on #177, which contains every commit here:
b07932f,10e9eb2,71ec418,8fa9a99,703dfdd,c179da2and7514c40fail without their fix.Known gaps, unchanged by this PR:
Which issue(s) this PR is related to:
None. The bugs reproduce on
mainand have no separate issues. #36 is fixed by #177.Special notes for your reviewer:
The commits are meant to be reviewed one by one.
a03f18cis the largest: it movesmain's matching code intonode_selector.gounchanged.Does this PR introduce a user-facing change?
🤖 Generated with Claude Code