[release-4.20] OCPBUGS-62273: Fix EgressIP stale GARP post reboot + pod restart #2767
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently, we are force exiting with the trap before the background processes can end, container is removed and the orphaned processes end early causing our config to go into an unknown state because we dont end in an orderly manner. Wait until the pid file for ovnkube controller with node is removed which shows the process has completed. Signed-off-by: Martin Kennelly <[email protected]> (cherry picked from commit 8b29419)
Prevent ovn-controller from sending stale GARP by adding drop flows on external bridge patch ports until ovnkube-controller synchronizes the southbound database - henceforth known as "drop flows". This addresses race conditions where ovn-controller processes outdated SB DB state before ovnkube-controller updates it, particularly affecting EIP SNAT configurations attached to logical router ports. Fixes: https://issues.redhat.com/browse/FDP-1537 ovnkube-controller controls the lifecycle of the drop flows. ovs / ovn-controller running is required to configure external bridge. Downstream, the external bridge maybe precreated and ovn-controller will use this. This fix considers three primary scenarios: node, container and pod restart. On Node restart means the ovs flows installed priotior to reboot on the node are cleared but the external bridge exists. Add the flows before ovnkube controller with node starts. The reason to add it here is that our gateway code depends on ovn-controller started and running... There is now a race here between ovn-controller starting (and garping) before we set this flow but I think the risk is low however it needs serious testing. The reason I did not naturally at the drop flows before ovn-controller started is because I have no way to detect if its a node reboot or pod reboot and i dont want to inject drop flows for simple ovn-controller container restart which could disrupt traffic. ovnkube-controller starts, we create a new gateway and apply flows the same flows in-order to ensure we always drop GARP when ovnkube controller hasn't sync. Remove the flows when ovnkube-controller has syncd. There is also a race here between ovnkube-controller removing the flows and ovn-controller GARPing with stale SB DB info. There is no easy way to detect what SB DB data ovn-controller has consumed. On Pod restart, we add the drop flows before exit. ovnkube-controller-with-node will also add it before it starts the go code. Container restart: - ovnkube-controller: adds flows upon start and exit - ovn-controller: no changes While the drop flows are set, OVN may not be able to resolve IPs it doesn't know about in its Logical Router pipelines generation. Following removal of the drop flows, OVN may resolve the IPs using GARP requests. OVN-Controller always sends out GARPs with op code 1 on startup. Signed-off-by: Martin Kennelly <[email protected]> (cherry picked from commit 82fc3bf)
PR 5373 to drop the GARP flows didnt consider that we set the default network controller and later we set the gateway obj. In-between this period, ovnkube node may receive a stop signal and we do not guard against accessing the gateway if its not yet set. OVNKube controller may have sync'd before the gateway obj is set. There is nothing to reconcile if the gateway is not set. Signed-off-by: Martin Kennelly <[email protected]> (cherry picked from commit e60220a)
openshift-ci
bot
added
the
do-not-merge/hold
openshift-ci-robot
added
jira/severity-important
|
@martinkennelly: This pull request references Jira Issue OCPBUGS-62273, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
bot
added
the
approved
|
/retest |
|
/payload 4.20 ci blocking |
|
@martinkennelly: trigger 5 job(s) of type blocking for the ci release of OCP 4.20
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/aff1c280-9ea3-11f0-9d4a-b7774b550ef6-0 trigger 10 job(s) of type blocking for the nightly release of OCP 4.20
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/aff1c280-9ea3-11f0-9d4a-b7774b550ef6-1 |
|
@martinkennelly: This PR was included in a payload test run from openshift/machine-config-operator#5317
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/c1411b20-9ea4-11f0-832b-31e3f9e7c09d-0 |
|
@martinkennelly: This PR was included in a payload test run from openshift/machine-config-operator#5317
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/d08c34c0-9ea4-11f0-9135-f805ed0e2cd9-0 |
|
/test e2e-gcp-ovn-techpreview Disruption seen: |
|
Lint maybe overridden https://issues.redhat.com/browse/CORENET-6207 |
|
@martinkennelly: This PR was included in a payload test run from openshift/machine-config-operator#5317
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/13c8aa30-9f85-11f0-8f90-4690ab5934ee-0 |
|
/test e2e-gcp-ovn-techpreview |
Ensure ovn-controller has processed the SB DB updates before removing the GARP drop flows by utilizing the hv_cfg field in NB_Global [1] OVNKube controller increments the nb_cfg value post sync, which is copied to SB DB by northd. OVN-Controllers copy this nb_cfg value from SB DB and write it to their chassis_private tables nb_cfg field after they have processed the SB DB changes. Northd will then look at all the chassis_private tables nb_cfg value and set the NB DBs Nb_global hv_cfg value to the min integer found. Since IC currently only supports one node per zone, we can be sure ovn-controller is running locally and therefore its ok to block removing the drop GARP flows. [1] https://man7.org/linux/man-pages/man5/ovn-nb.5.html Signed-off-by: Martin Kennelly <[email protected]> (cherry picked from commit 3b5da01)
|
@martinkennelly: This PR was included in a payload test run from openshift/machine-config-operator#5317
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/3c9dcbe0-a43d-11f0-8492-d57d926fbb48-0 |
|
@martinkennelly: This PR was included in a payload test run from openshift/machine-config-operator#5317
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/5a2d3d80-a43d-11f0-8069-b32b778be59b-0 |
|
/test e2e-aws-ovn-local-to-shared-gateway-mode-migration Known issue: |
|
/test e2e-aws-ovn-upgrade Unrelated - couldnt find a bug. |
|
/test e2e-azure-ovn-upgrade |
|
/test e2e-gcp-ovn Same error sig outside this PR: Also deprovisioning failed: Theres are unrelated to this PR |
|
/test e2e-gcp-ovn-techpreview Unrelated - no bug found: |
openshift-ci
bot
assigned
jechen0648,
Meina-rh,
mffiedler,
qiowang721,
rbbratta and
weliang1
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jcaamano, martinkennelly The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/jira refresh |
|
@martinkennelly: This pull request references Jira Issue OCPBUGS-62273, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
|
@martinkennelly: This pull request references Jira Issue OCPBUGS-62273, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
openshift-ci-robot
added
the
jira/valid-bug
|
@martinkennelly: This pull request references Jira Issue OCPBUGS-62273, which is valid. 9 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Jira ([email protected]), skipping review request. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
removed
the
jira/invalid-bug
|
Waiting on perf scale team to ack / not ack the CI resules for job |
|
@jcaamano ^ they said its nothing to do with my PR. |
|
/test |
|
@jtaleric: The The following commands are available to trigger optional jobs: Use In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test qe-perfscale-payload-control-plane-6nodes |
|
/override ci/prow/lint |
|
@jcaamano: Overrode contexts on behalf of jcaamano: ci/prow/lint In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
No need to retry if we dont think its a problem with this PR - its only going to waste time. |
|
/label staff-eng-approved |
openshift-ci
bot
added
the
staff-eng-approved
|
/unhold |
openshift-ci
bot
removed
the
do-not-merge/hold
openshift-merge-bot
bot
merged commit 84cdb99
into
openshift:release-4.20
28 of 29 checks passed
|
@martinkennelly: Jira Issue Verification Checks: Jira Issue OCPBUGS-62273 Jira Issue OCPBUGS-62273 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
/hold