Skip to content

[shiftstack] Bootstrap oc when prepare stage is skipped - #4195

Closed
tusharjadhav3302 wants to merge 1 commit into
openstack-k8s-operators:mainfrom
tusharjadhav3302:fix/shiftstack-bootstrap-oc-when-prepare-skipped
Closed

tusharjadhav3302 wants to merge 1 commit into
openstack-k8s-operators:mainfrom
tusharjadhav3302:fix/shiftstack-bootstrap-oc-when-prepare-skipped

Conversation

@tusharjadhav3302

Copy link
Copy Markdown
Contributor

Summary

Phased Pipeline B (run-tests) sets cifmw_shiftstack_stages_override to verification/test stages only and omits prepare. The shiftstack role always recreates the shiftstackclient pod, and QA only installs /usr/local/bin/oc during prepare. Result: verification fails immediately with oc: command not found.

When stages_override is non-empty and does not include prepare, bootstrap a stable OpenShift client into the pod (same cold-start approach as shiftstack-qa get_openshift_release_binaries) before running tests.

  • New task: roles/shiftstack/tasks/bootstrap_oc_client.yml
  • Gated so monolithic jobs (stages_override: []) are unchanged
  • Optional kill-switch: cifmw_shiftstack_bootstrap_oc_client: false

Test plan

  • Manual on warm serval70: recreate shiftstackclient pod → confirm oc missing → run bootstrap → oc on PATH (Zuul failure mode gone)
  • Zuul check on this PR
  • Follow-up: Depends-On from Pipeline B TP / jobs MR (not updated in this change)

Made with Cursor

@openshift-ci

openshift-ci Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign eurijon for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@openshift-ci

openshift-ci Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Hi @tusharjadhav3302. Thanks for your PR.

I'm waiting for a openstack-k8s-operators member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-bootstrap-oc-when-prepare-skipped branch from 01becc7 to 0238c9c Compare September 21, 2026 07:15
@brjackma

Copy link
Copy Markdown
Contributor

/ok-to-test

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/92584dceeda64dd9a1cd7bcc125381b2

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 37m 21s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 11s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 34m 48s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 01m 24s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 33s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 43m 07s
❌ cifmw-crc-podified-edpm-baremetal-bootc FAILURE in 33m 05s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 04s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 48s
✔️ cifmw-molecule-shiftstack SUCCESS in 23m 39s

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/30aa43bfd4e24d26b0c01699d1621345

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 11m 01s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 24m 26s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 39m 14s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 54m 46s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 30s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 1h 00m 19s
❌ cifmw-crc-podified-edpm-baremetal-bootc NODE_FAILURE Node(set) request 099-0000209262 failed in 0s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 38s
✔️ cifmw-pod-pre-commit SUCCESS in 7m 55s
✔️ cifmw-molecule-shiftstack SUCCESS in 25m 24s

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

/retest-required

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/aa0abdd147914eb5aee3371ae686534f

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 54m 30s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 24m 18s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 31m 04s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 02m 32s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 26s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 3h 53m 31s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 26m 54s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 19s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 19s
✔️ cifmw-molecule-shiftstack SUCCESS in 24m 32s

@michburk

Copy link
Copy Markdown
Contributor

Hi, I think we on the framework team might be missing a bit of context from the summary you provided. I'm not certain what phased pipeline b is, but after doing quite a bit of digging it looks like this pr is trying to address a situation that arose because of the way some job was split into several jobs.

To my understanding, it looks like there used to be one shiftstackclient pod created and used across a single monolithic test suite, but those tests have now been broken up into multiple phases. Now a new shiftstackclient pod is created several times, so some amount of configuration (oc client install, /etc/hosts entries, etc) needs to be put in all the right places in the new pod before testing can resume.

Correct me if my understanding is incorrect on any of the above

This leaves me wondering, is this something that could be handled from the shiftstack qa side? For example, could there be a resume stage or something? It strikes me that if the shiftstack qa repo is already 'familiar with' how to prepare the environment inside the pod, then subsequent logic doing similar operations would also belong in shiftstack qa, not the cifmw. Again I'm very unfamiliar with all of the context here, so please pardon my ignorance.

Additionally, if this does need to live in the cifmw, I will ask that you maintain a clean git history. We don't squash commits when we merge, so instead of committing something broken then immediately fixing it with a second commit, just amend the initial commit.

Thanks!

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

Hi @michburk — thanks for digging in. Your reading is mostly correct; a bit more context:

What changed
We split the ShiftStack nightly into two Zuul pipelines. Pipeline B (Workload Verification) is three jobs: platform-precheck → deploy-ocp-guest → run-tests. The run-tests job only runs verification/test stages via cifmw_shiftstack_stages_override and omits prepare, so we do not redo guest install/setup that already ran in the previous job.

Why oc is missing
Each job that invokes the cifmw shiftstack role creates a new shiftstackclient pod. QA installs /usr/local/bin/oc only in the prepare stage. With prepare skipped, that fresh pod has no oc, and verification fails immediately with oc: command not found. Monolithic jobs are unaffected (stages_override: [] → this bootstrap does not run).

Why this lives in cifmw (not a QA resume stage)

  1. Pod recreate is owned by the cifmw shiftstack role, before QA stages run.
  2. Skipping prepare is a cifmw/Zuul contract for Pipeline B.
  3. This PR only cold-starts oc into the new pod when cifmw has already chosen to omit prepare — same approach QA uses in prepare, scoped so monolithic behavior stays unchanged.

A QA resume stage would still need something to install oc after every pod recreate; putting that behind the cifmw stages_override gate keeps the fix next to the contract that skips prepare.

Happy to move the bootstrap into shiftstack-qa later if that team prefers ownership there; this PR unblocks Pipeline B without changing the monolithic path.

On history: we will amend / leave a clean single commit for merge (no broken-then-fix pair).

Phased Pipeline B run-tests recreates the shiftstackclient pod but omits
the QA prepare stage. Without prepare, verification fails on missing oc,
/etc/hosts FIP entries, clouds.shiftstack, and artifacts/resources.yml.

When stages_override is set and does not include prepare, restore that
cold-start environment from the installation PVC and install-config
(same sources prepare uses) before running tests. Use a literal block
scalar for the in-pod shell so newlines are preserved.

Signed-off-by: Tushar Jadhav <tjadhav@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-bootstrap-oc-when-prepare-skipped branch from f60fe4a to ff2eca5 Compare September 23, 2026 07:50
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/c9e14a1248f2420082a6d21acbea8ce7

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 20m 37s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 24m 49s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 34m 02s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 54m 31s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 39s
❌ openstack-k8s-operators-content-provider-bootc FAILURE in 19m 05s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 16s
✔️ cifmw-pod-pre-commit SUCCESS in 9m 05s
✔️ cifmw-molecule-shiftstack SUCCESS in 15m 31s

@michburk

Copy link
Copy Markdown
Contributor

Thanks for cleaning up the git history, I'm still a little unclear on why this logic needs to live in the cifmw.

To these two points:

cifmw has already chosen to omit prepare

and

putting that behind the cifmw stages_override gate keeps the fix next to the contract that skips prepare

The instruction to skip the prepare step comes from the job configuration, no? That is, the job sets the cifmw_shiftstack_stages_override list, the ci-framework passes this list of stages to shiftstack qa, and shiftstack qa runs the requested stages. Unless I'm mistaken on this, I don't see how the cifmw has 'chosen to omit prepare' and why it is the cifmw's responsibility to take on the configuration that the shiftstack qa repo normally does.

And to this point:

Happy to move the bootstrap into shiftstack-qa later if that team prefers ownership there;

I noticed you were active in the shiftstack qa repo and you are listed as an approver/reviewer there, which is largely why I'm asking you if this logic is better handled in that repo, sorry if I'm misunderstanding your role there.

I'm mostly concerned with not duplicating code across repos. If you change anything about the configurations in the prepare step of the shiftstack qa repo, the logic here in the cifmw would diverge. To me, it would make more sense to track all of that in one repo, and the shiftstack qa repo already owns similar logic in the prepare stage.

And again I'm by no means an expert on the shiftstack qa repo, I've just glanced over it to understand the gist of how things work. Is there some concrete technical reason why something like a resume stage couldn't exist there? You say

A QA resume stage would still need something to install oc after every pod recreate

but isn't a similar task already handled by the prepare stage?

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

Hi @michburk — thanks again for the careful review and for pushing on ownership/duplication. You were right.

The stages_override list is set by the jobs config and passed through cifmw into shiftstack-qa, so the warm-path rehydrate (install oc, restore clouds//etc/hosts/resources.yml without re-running full prepare) belongs next to the existing prepare logic in shiftstack-qa, not duplicated in cifmw.

We moved that work into shiftstack-qa as a dedicated stage:

Callers that omit prepare can include prepare_client_pod in the stages list before verification. That keeps a single source of truth with the prepare-owned tooling and avoids cifmw diverging if prepare changes.

We validated Pipeline B on warm serval70 without this PR (no Depends-On on #4195; CIF stayed on main). platform-precheck and deploy-ocp-guest succeeded; run-tests reached the verification suite (POST_FAILURE was an OTE suite failure, not the earlier oc: command not found bootstrap gap).

Closing this PR in favor of shiftstack-qa#43. Appreciate you taking the time to dig through the phased-job context.

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

Closing in favor of shiftstack-qa#43 (prepare_client_pod). See reply to @michburk above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants