Use a different minimal case

[stbenjam]

case-001 is the "hard" case Opus always fails at, case-003 is a better smoke test to make sure things are working.

Summary by CodeRabbit

This PR updates the OpenShift CI configuration for the openshift-eng/ai-helpers repository to improve the reliability of its evaluation smoke test.

Change: The eval-payload-analysis-minimal test's EVAL_CASES environment variable is changed from case-001 to case-003. This minimal test is a fast-feedback smoke test that runs evaluation workloads with reduced scope (250 max turns instead of 2500) to quickly verify the system is functioning.

Rationale: The PR replaces case-001, which is identified as a hard test case that consistently fails when evaluated with Claude Opus, with case-003, which serves as a more reliable baseline for verifying system functionality. This change ensures the smoke test doesn't get blocked on a known problematic test case and can provide useful early feedback during CI runs.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 0a86daf2-b218-45d8-b2d0-86e6f52a03a8

📥 Commits

Reviewing files that changed from the base of the PR and between b3e0d71 and 76c1044.

📒 Files selected for processing (1)

• ci-operator/config/openshift-eng/ai-helpers/openshift-eng-ai-helpers-main.yaml

---

Walkthrough

The EVAL_CASES environment variable in the eval-payload-analysis test within openshift-eng-ai-helpers-main.yaml is updated from case-001 to case-003. No other configuration values are modified.

Changes

CI Config Update

Layer / File(s)	Summary
Update EVAL_CASES env var ci-operator/config/openshift-eng/ai-helpers/openshift-eng-ai-helpers-main.yaml	EVAL_CASES value changed from case-001 to case-003 in the eval-payload-analysis test entry.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Use a different minimal case' directly corresponds to the change: switching the EVAL_CASES from case-001 to case-003. It clearly summarizes the primary change.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	Pull request modifies only CI operator config YAML file, not Ginkgo test code. Check for stable test names is not applicable to this configuration-only change.
Test Structure And Quality	✅ Passed	The PR only modifies a CI configuration YAML file (environment variable change), not Ginkgo test code. The custom check for test structure and quality is not applicable to configuration files.
Microshift Test Compatibility	✅ Passed	This PR does not add any new Ginkgo e2e tests. It only modifies a CI operator configuration file that changes an environment variable (EVAL_CASES) from case-001 to case-003. The MicroShift compatib...
Single Node Openshift (Sno) Test Compatibility	✅ Passed	PR modifies a CI configuration YAML file only, not adding new Ginkgo e2e tests; check is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	PR modifies a CI test configuration file, not deployment manifests, operator code, or controllers. The check explicitly applies only to those artifacts; this change introduces no scheduling constra...
Ote Binary Stdout Contract	✅ Passed	PR only modifies a YAML CI configuration file (changing EVAL_CASES from case-001 to case-003), not any OTE binary source code. Check is not applicable.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PR does not add new Ginkgo e2e tests; it only changes an environment variable in a CI configuration file for an AI helpers evaluation workflow, which is outside the scope of this check.
No-Weak-Crypto	✅ Passed	PR only modifies CI operator YAML configuration, changing a test case parameter. No weak cryptographic algorithms (MD5, SHA1, DES, RC4, 3DES, Blowfish, ECB), custom crypto implementations, or non-c...
Container-Privileges	✅ Passed	The PR changes only an environment variable (EVAL_CASES) in a CI operator config file. No privileged container settings, capabilities, or escalation configurations are introduced or modified.
No-Sensitive-Data-In-Logs	✅ Passed	The PR changes only the EVAL_CASES test identifier from case-001 to case-003 in a CI config file. No sensitive data like passwords, tokens, API keys, PII, or credentials are exposed in the logs.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-merge-bot]

[REHEARSALNOTIFIER]
@stbenjam: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-eng-ai-helpers-main-eval-payload-analysis-minimal	openshift-eng/ai-helpers	presubmit	Ci-operator config changed

Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[openshift-ci]

@stbenjam: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[stbenjam]

/pj-rehearse ack

[openshift-merge-bot]

@stbenjam: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: petr-muller, stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• ci-operator/config/openshift-eng/ai-helpers/OWNERS [stbenjam]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment