Skip to content

OCPBUGS-84695: [TNF] Fix update-setup job discovery to limit check to active job - #31451

Merged
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
jaypoulz:node-replacement-fix-84695
Jul 30, 2026
Merged

OCPBUGS-84695: [TNF] Fix update-setup job discovery to limit check to active job#31451
openshift-merge-bot[bot] merged 1 commit into
openshift:mainfrom
jaypoulz:node-replacement-fix-84695

Conversation

@jaypoulz

@jaypoulz jaypoulz commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Wait for single update-setup job on survivor node only
The survivor node's update-setup job performs the actual Pacemaker reconciliation in both old and new CEO architectures:

  • Old CEO: creates 2 per-node jobs; target exits early, survivor works
  • New CEO: creates 1 cluster-wide job with round-robin retry (always succeeds on survivor during replacement because Pacemaker runs there)

Remove the wait for target-node job since it either doesn't exist (new) or does nothing (old). Only the survivor job matters.

Summary by CodeRabbit

Summary by CodeRabbit

  • Tests
    • Updated cluster restoration validation to consider only the survivor node’s reconciliation (“update-setup”) job for completion.
    • Adjusted the timing expectations used during restoration to allow a longer reconciliation window.
    • Removed validation tied to a separate target-node reconciliation job and clarified survivor readiness timing constraints.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 29, 2026
@openshift-ci
openshift-ci Bot requested review from fracappa and jeff-roche July 29, 2026 21:13
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 5d36bb09-afee-47db-9a1f-9355b1033b53

📥 Commits

Reviewing files that changed from the base of the PR and between 24d8186 and 6590e15.

📒 Files selected for processing (2)
  • test/extended/edge_topologies/tnf_node_replacement_const.go
  • test/extended/edge_topologies/tnf_node_replacement_finish.go

Walkthrough

restorePacemakerCluster now waits only for the survivor node’s CEO update-setup job, retains nodeReadyTime as the minimum pod creation constraint, and uses a five-minute wait timeout.

Changes

Pacemaker cluster restoration

Layer / File(s) Summary
Survivor-only update-setup reconciliation
test/extended/edge_topologies/tnf_node_replacement_finish.go, test/extended/edge_topologies/tnf_node_replacement_const.go
The restoration flow removes target-job synchronization, retains the survivor completion assertion, and increases the update-setup wait timeout from three to five minutes.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: fracappa, jeff-roche


Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

  • Ignore

❌ Failed checks (1 error, 1 warning)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error The patch adds e2e.Logf(...survivor node %s...), exposing an internal node name in logs. Remove or anonymize node/IP identifiers from new log messages, or gate them behind a redaction-safe debug path.
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (13 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title matches the core change: update-setup job handling now checks only the active survivor job and drops the target-job wait.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed Touched files only adjust helper logic and a timeout; no test titles were changed, and existing Ginkgo titles in the package are static.
Test Structure And Quality ✅ Passed The changed Ginkgo flow has BeforeEach/AfterEach cleanup, bounded waits, and all assertions include messages; the survivor-job waiter also has an explicit timeout.
Microshift Test Compatibility ✅ Passed No new Ginkgo tests were added; the TNF spec is already tagged [apigroup:config.openshift.io], so MicroShift auto-skips it.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The TNF spec already skips unless control-plane topology is DualReplica, so it won’t run on SNO; no new unguarded Ginkgo test was added.
Topology-Aware Scheduling Compatibility ✅ Passed Only test waiting logic and a timeout changed; no manifests, controllers, affinity, node selectors, replicas, or topology assumptions were introduced.
Ote Binary Stdout Contract ✅ Passed The PR only changes job-wait logic and a timeout constant; the touched files contain no process-level stdout writes, init/main/TestMain, or suite setup output setup.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed Only job-wait timing changed; no new Ginkgo test or IPv4/public-internet dependency was added, and BMC URL building already uses net.JoinHostPort.
No-Weak-Crypto ✅ Passed Diff only changes job-wait logic/timeouts; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons appear.
Container-Privileges ✅ Passed Diff only changes two Go test files; no K8s/container manifests or privilege fields (privileged, hostPID/Network/IPC, SYS_ADMIN, allowPrivilegeEscalation, root) were added.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jul 29, 2026
@jaypoulz
jaypoulz force-pushed the node-replacement-fix-84695 branch 2 times, most recently from d919611 to 6c7848d Compare July 29, 2026 21:25
@openshift-ci

openshift-ci Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

@jaypoulz: This PR was included in a payload test run from openshift/cluster-etcd-operator#1655
trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-1of3
  • periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-2of3
  • periodic-ci-openshift-release-main-nightly-4.22-e2e-metal-ovn-two-node-fencing-recovery-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/34b294b0-8b96-11f1-9903-4bbed5998a73-0

@openshift-ci

openshift-ci Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

@jaypoulz: This PR was included in a payload test run from openshift/cluster-etcd-operator#1655
trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ovn-two-node-fencing-recovery-1of3
  • periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ovn-two-node-fencing-recovery-2of3
  • periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ovn-two-node-fencing-recovery-3of3

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/52a7cd00-8b96-11f1-9573-a9b6be14b37d-0

@jaypoulz
jaypoulz force-pushed the node-replacement-fix-84695 branch from 6c7848d to 24d8186 Compare July 29, 2026 21:48
@jaypoulz

Copy link
Copy Markdown
Contributor Author

@coderabbitai I'm not addressing the warning because it's not adding new logs, nor is the internal IP of a virtualized node a secret worth protecting. Please mark this as ignored.

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

gh api graphql \
  -F owner="openshift" \
  -F name="origin" \
  -F number=31451 \
  -f query='
query($owner: String!, $name: String!, $number: Int!) {
  repository(owner: $owner, name: $name) {
    pullRequest(number: $number) {
      reviewThreads(first: 100) {
        nodes {
          id
          isResolved
          path
          line
          comments(first: 20) {
            nodes {
              author { login }
              body
              url
            }
          }
        }
      }
    }
  }
}' \
  --jq '
    .data.repository.pullRequest.reviewThreads.nodes[]
    | select(any(.comments.nodes[]?; .author.login == "coderabbitai"))
    | {
        id,
        isResolved,
        path,
        line,
        comments: [
          .comments.nodes[]
          | select(.author.login == "coderabbitai")
          | {author: .author.login, body, url}
        ]
      }'

Length of output: 1061


🏁 Script executed:

#!/bin/bash
set -euo pipefail

repo="openshift/origin"
pr="31451"
pattern='(?i)(\blog(s|ging)?\b|\bip\b|address|secret|sensitive|redact|internal)'

echo "== Inline review comments =="
gh api --paginate "repos/$repo/pulls/$pr/comments?per_page=100" \
  --jq ".[] | select((.body // \"\") | test(\"$pattern\")) | {
    id, node_id, path, line, author: .user.login, body, html_url
  }"

echo "== Pull-request reviews =="
gh api --paginate "repos/$repo/pulls/$pr/reviews?per_page=100" \
  --jq ".[] | select((.body // \"\") | test(\"$pattern\")) | {
    id, node_id, state, author: .user.login, body, html_url
  }"

Length of output: 805


🏁 Script executed:

#!/bin/bash
set -euo pipefail

gh api --paginate "repos/openshift/origin/issues/31451/comments?per_page=100" \
  --jq '
    .[]
    | select((.user.login // "") | test("coderabbit"; "i"))
    | {
        id,
        node_id,
        author: .user.login,
        body,
        html_url,
        created_at
      }'

Length of output: 13465


@jaypoulz I couldn’t mark the warning ignored because no matching unresolved CodeRabbit review thread is currently present on #31451. The available CodeRabbit review reports no actionable comments and marks the sensitive-data-in-logs check as passed.

I’ve recorded the clarification for future reviews.


✏️ Learnings added
Learnt from: jaypoulz
URL: https://github.com/openshift/origin/pull/31451

Timestamp: 2026-07-29T21:51:58.117Z
Learning: For OpenShift virtualized-node test logging, an existing internal IP address is not considered secret or sensitive data. Do not flag it as a new sensitive-data-in-logs concern when a change does not add logging.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

@jaypoulz jaypoulz changed the title Fix update-setup job discovery for cluster-wide jobs NO-JIRA: [TNF] Fix update-setup job discovery for cluster-wide jobs Jul 29, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 29, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@jaypoulz: This pull request explicitly references no jira issue.

Details

In response to this:

Wait for single update-setup job on survivor node only
The survivor node's update-setup job performs the actual Pacemaker reconciliation in both old and new CEO architectures:

  • Old CEO: creates 2 per-node jobs; target exits early, survivor works
  • New CEO: creates 1 cluster-wide job with round-robin retry (always succeeds on survivor during replacement because Pacemaker runs there)

Remove the wait for target-node job since it either doesn't exist (new) or does nothing (old). Only the survivor job matters.

Summary by CodeRabbit

  • Tests
  • Updated cluster restoration validation to wait only for the survivor node’s reconciliation (“update-setup”) job to complete successfully.
  • Improved validation coverage for both per-node and cluster-wide reconciliation flows.
  • Removed validation that depended on completion of a separate target-node reconciliation job, and tightened the survivor readiness constraint.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@jaypoulz jaypoulz changed the title NO-JIRA: [TNF] Fix update-setup job discovery for cluster-wide jobs NO-JIRA: [TNF] Fix update-setup job discovery to limit check to active job Jul 29, 2026
@jaypoulz jaypoulz changed the title NO-JIRA: [TNF] Fix update-setup job discovery to limit check to active job OCPBUGS-84695: [TNF] Fix update-setup job discovery to limit check to active job Jul 29, 2026
@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Jul 29, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@jaypoulz: This pull request references Jira Issue OCPBUGS-84695, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Wait for single update-setup job on survivor node only
The survivor node's update-setup job performs the actual Pacemaker reconciliation in both old and new CEO architectures:

  • Old CEO: creates 2 per-node jobs; target exits early, survivor works
  • New CEO: creates 1 cluster-wide job with round-robin retry (always succeeds on survivor during replacement because Pacemaker runs there)

Remove the wait for target-node job since it either doesn't exist (new) or does nothing (old). Only the survivor job matters.

Summary by CodeRabbit

  • Tests
  • Updated cluster restoration validation to wait only for the survivor node’s reconciliation (“update-setup”) job to complete successfully.
  • Improved validation coverage for both per-node and cluster-wide reconciliation flows.
  • Removed validation that depended on completion of a separate target-node reconciliation job, and tightened the survivor readiness constraint.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-metal-ovn-two-node-arbiter
/test e2e-metal-ovn-two-node-fencing
/test e2e-metal-ovn-two-node-fencing-recovery

The survivor node's update-setup job performs the actual Pacemaker
reconciliation in both old and new CEO architectures:
- Old CEO: creates 2 per-node jobs; target exits early, survivor works
- New CEO: creates 1 cluster-wide job with round-robin retry (always
  succeeds on survivor during replacement because Pacemaker runs there)

Remove the wait for target-node job since it either doesn't exist (new)
or does nothing (old). Only the survivor job matters.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@jaypoulz
jaypoulz force-pushed the node-replacement-fix-84695 branch from 24d8186 to 6590e15 Compare July 30, 2026 12:35
@jaypoulz

Copy link
Copy Markdown
Contributor Author

Bumped the timeout from 3 to 5 minute since it didn't quite make it on the 5.0 verification, but the 4.22 verification passed.

@jaypoulz

Copy link
Copy Markdown
Contributor Author

/verified by #31451 (comment) shard 2

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 30, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@jaypoulz: This PR has been marked as verified by https://github.com/openshift/origin/pull/31451#issuecomment-5123653274 shard 2.

Details

In response to this:

/verified by #31451 (comment) shard 2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot

Copy link
Copy Markdown

@jaypoulz: This pull request references Jira Issue OCPBUGS-84695, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

Wait for single update-setup job on survivor node only
The survivor node's update-setup job performs the actual Pacemaker reconciliation in both old and new CEO architectures:

  • Old CEO: creates 2 per-node jobs; target exits early, survivor works
  • New CEO: creates 1 cluster-wide job with round-robin retry (always succeeds on survivor during replacement because Pacemaker runs there)

Remove the wait for target-node job since it either doesn't exist (new) or does nothing (old). Only the survivor job matters.

Summary by CodeRabbit

Summary by CodeRabbit

  • Tests
  • Updated cluster restoration validation to consider only the survivor node’s reconciliation (“update-setup”) job for completion.
  • Adjusted the timing expectations used during restoration to allow a longer reconciliation window.
  • Removed validation tied to a separate target-node reconciliation job and clarified survivor readiness timing constraints.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@fracappa fracappa left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 30, 2026
@openshift-ci

openshift-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fracappa, jaypoulz, vimauro

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-metal-ovn-two-node-arbiter
/test e2e-metal-ovn-two-node-fencing
/test e2e-metal-ovn-two-node-fencing-recovery

@jaypoulz

Copy link
Copy Markdown
Contributor Author

/retest-required

@openshift-merge-bot
openshift-merge-bot Bot merged commit 9a3328c into openshift:main Jul 30, 2026
22 of 25 checks passed
@openshift-ci-robot

Copy link
Copy Markdown

@jaypoulz: Jira Issue OCPBUGS-84695: Some pull requests linked via external trackers have merged:

The following pull request, linked via external tracker, has not merged:

All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-84695 has not been moved to the MODIFIED state.

This PR is marked as verified. If the remaining PRs listed above are marked as verified before merging, the issue will automatically be moved to VERIFIED after all of the changes from the PRs are available in an accepted nightly payload.

Details

In response to this:

Wait for single update-setup job on survivor node only
The survivor node's update-setup job performs the actual Pacemaker reconciliation in both old and new CEO architectures:

  • Old CEO: creates 2 per-node jobs; target exits early, survivor works
  • New CEO: creates 1 cluster-wide job with round-robin retry (always succeeds on survivor during replacement because Pacemaker runs there)

Remove the wait for target-node job since it either doesn't exist (new) or does nothing (old). Only the survivor job matters.

Summary by CodeRabbit

Summary by CodeRabbit

  • Tests
  • Updated cluster restoration validation to consider only the survivor node’s reconciliation (“update-setup”) job for completion.
  • Adjusted the timing expectations used during restoration to allow a longer reconciliation window.
  • Removed validation tied to a separate target-node reconciliation job and clarified survivor readiness timing constraints.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

@jaypoulz: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-metal-ovn-two-node-fencing 6590e15 link false /test e2e-metal-ovn-two-node-fencing

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@jaypoulz

Copy link
Copy Markdown
Contributor Author

/cherry-pick release-4.22

@openshift-cherrypick-robot

Copy link
Copy Markdown

@jaypoulz: new pull request created: #31467

Details

In response to this:

/cherry-pick release-4.22

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants