CMP-3824: Added logic to cover 27967, 33782, 33711, 47346 from downstream -WIP by taimurhafeez · Pull Request #1071 · ComplianceAsCode/compliance-operator

taimurhafeez · 2026-01-27T14:16:53Z

No description provided.

openshift-ci · 2026-01-27T14:17:06Z

Hi @taimurhafeez. Thanks for your PR.

I'm waiting for a ComplianceAsCode member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci-robot · 2026-01-27T14:28:59Z

@taimurhafeez: This pull request references CMP-3824 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

tests/e2e/serial/main_test.go

openshift-ci · 2026-02-06T11:36:03Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: taimurhafeez

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [taimurhafeez]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

github-actions · 2026-02-06T11:42:50Z

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1071-a740d5362a529a7316c683e2bcabe2765d2655b1

github-actions · 2026-02-09T16:08:39Z

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1071-a8c3cf4c57dd6fb0d0e14bc501627dc7fb3c0913

taimurhafeez · 2026-02-25T13:11:47Z

/test e2e-aws-serial-arm

# Conflicts: # tests/e2e/framework/common.go

github-actions · 2026-03-05T11:44:28Z

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1071-c360aca9a9e80e8c09a443e677b1e4ddf743d931

github-actions · 2026-03-05T11:47:21Z

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:1071-ef10ef564bc60531d3b83f9a33cd2927e9399fcb

rhmdnd

Some comments inline. Thanks!

rhmdnd · 2026-03-05T17:33:16Z

tests/e2e/serial/main_test.go

 	if err != nil {
-		t.Fatal(err)
+		// If it's not NON-COMPLIANT, check if it's COMPLIANT (all rules already compliant)
+		err2 := f.WaitForSuiteScansStatus(f.OperatorNamespace, suiteName, compv1alpha1.PhaseDone, compv1alpha1.ResultCompliant)


Looks like we need that PR that adds multiple results to the WaitForSuiteScansStatus do avoid the double loop.

@Anna-Koudelkova is adding that in another testing PR, and I think you are too. I wonder if we should just pull that out so we can merge that and then rebase.

Otherwise, in the worst case we wait the full time out for a compliance result.

rhmdnd · 2026-03-05T17:37:19Z

tests/e2e/serial/main_test.go

+	// This allows the test to work regardless of which rules are non-compliant in the environment
+	remediationList := &compv1alpha1.ComplianceRemediationList{}
+	listOpts := client.ListOptions{
+		Namespace: f.OperatorNamespace,


Could add additional filtering here to only get remediations for the suite in question, but not a real big deal since this is serial and there should only be one suite running at a time.

listOpts := client.ListOptions{ Namespace: f.OperatorNamespace, LabelSelector: labels.SelectorFromSet(map[string]string{compv1alpha1.SuiteLabel: suiteName}), }

Completely up to you, it shouldn't affect the test.

rhmdnd · 2026-03-05T17:38:07Z

tests/e2e/serial/main_test.go

+			t.Fatal(err)
+		}
+		appliedRemediations = append(appliedRemediations, updatedRem)
+		log.Printf("✓ Remediation %d/%d applied successfully: %s\n", i+1, len(ourRemediations), rem.Name)


We don't usually put emojis in test logging. Could refactor this out.

rhmdnd · 2026-03-05T17:39:19Z

tests/e2e/serial/main_test.go

+		if checkResult.Status != compv1alpha1.CheckResultPass {
+			t.Fatalf("Expected check %s to PASS after remediation, got %s", checkName, checkResult.Status)
+		}
+		log.Printf("✓ Check %s is PASS\n", checkName)


Similar comment here.

rhmdnd · 2026-03-05T17:42:33Z

tests/e2e/serial/main_test.go

-			if mc.Name == rem.GetMcName() {
-				return false, nil
+
+	for i, rem := range appliedRemediations {


Cleaning up the machine configs pools is required for node remediations, but what about any rules that are using API server remediations?

Are we ensuring the remediations we're applying are only using machine configs? I'm wondering if we need to implement something to make sure API server remediations also get unapplied before the next test case runs.

Actually - looking at the tailored profile, we're only including three RHCOS4 rules, which should be fine for this since those have to use machine configs for remediation.

rhmdnd · 2026-03-05T17:43:26Z

tests/e2e/serial/main_test.go

+					Rationale: "Test file-based remediation",
+				},
+				{
+					// Issue 47346: Array value remediation


Irrelevant test ID?

rhmdnd · 2026-03-05T17:43:31Z

tests/e2e/serial/main_test.go

+					Rationale: "Test array value remediation with multiple NTP servers",
+				},
+				{
+					// Issue 27967: Different remediation type (audit rules)


Irrelevant test ID?

rhmdnd · 2026-03-05T17:43:46Z

tests/e2e/serial/main_test.go

+			t.Fatal(err)
+		}
+
+		// Issue 33711: Verify ComplianceRemediation shows "Applied" status


Irrelevant test ID?

rhmdnd · 2026-03-05T17:43:53Z

tests/e2e/serial/main_test.go

-	if err != nil {
-		t.Fatal(err)
-	}
+	// Issue 33711: Verify all checks pass after remediation (Applied → PASS transition)


Irrelevant test ID?

openshift-ci · 2026-03-05T19:02:39Z

@taimurhafeez: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-rosa	`ef10ef5`	link	true	`/test e2e-rosa`
ci/prow/e2e-aws-parallel	`ef10ef5`	link	true	`/test e2e-aws-parallel`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot added do-not-merge/work-in-progress needs-ok-to-test labels Jan 27, 2026

taimurhafeez changed the title ~~Added logic to cover 27967, 33782, 33711, 47346 from downstream~~ CMP-3824: Added logic to cover 27967, 33782, 33711, 47346 from downstream Jan 27, 2026

openshift-ci-robot added the jira/valid-reference label Jan 27, 2026

taimurhafeez changed the title ~~CMP-3824: Added logic to cover 27967, 33782, 33711, 47346 from downstream~~ CMP-3824: Added logic to cover 27967, 33782, 33711, 47346 from downstream -WIP Jan 27, 2026

xiaojiey reviewed Feb 6, 2026

View reviewed changes

tests/e2e/serial/main_test.go Outdated Show resolved Hide resolved

taimurhafeez marked this pull request as ready for review February 6, 2026 11:33

openshift-ci bot removed the do-not-merge/work-in-progress label Feb 6, 2026

openshift-ci bot requested review from jhrozek and rhmdnd February 6, 2026 11:33

openshift-ci bot added the approved label Feb 6, 2026

taimurhafeez added 5 commits March 5, 2026 11:38

Added logic to cover 27967, 33782, 33711, 47346 from downstream

43b52f8

copied back common file from master to replace changed caused by PR 960

04783d6

Modified timeout values to see if CI jobs are successful.

9a9128e

Removed IDs

1f56c5c

Merge remote-tracking branch 'origin/master' into CMP-3824

ef10ef5

# Conflicts: # tests/e2e/framework/common.go

taimurhafeez force-pushed the CMP-3824 branch from c360aca to ef10ef5 Compare March 5, 2026 11:39

rhmdnd added the ok-to-test label Mar 5, 2026

rhmdnd requested changes Mar 5, 2026

View reviewed changes

openshift-ci bot assigned rhmdnd Mar 5, 2026

Conversation

taimurhafeez commented Jan 27, 2026

Uh oh!

openshift-ci bot commented Jan 27, 2026

Uh oh!

openshift-ci-robot commented Jan 27, 2026

Uh oh!

Uh oh!

openshift-ci bot commented Feb 6, 2026

Uh oh!

github-actions bot commented Feb 6, 2026

Uh oh!

github-actions bot commented Feb 9, 2026

Uh oh!

taimurhafeez commented Feb 25, 2026

Uh oh!

github-actions bot commented Mar 5, 2026

Uh oh!

github-actions bot commented Mar 5, 2026

Uh oh!

rhmdnd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Mar 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants