-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ability to waive
non-blocking failures in tests
#2422
Comments
Hi @lsm5 ! We've shortly discussed this within a team and have a few questions to properly understand the use case:
|
Thanks!
Not sure what you mean by
Ack, is it possible to mark a feature as GitHub-only?
Thanks for sharing that about TMT. Didn't know this.
Hmm, maybe both. Let's say I review 10 PRs in a day and I see the same failure in the first 2-3 and decide to waive them. It would be convenient to skip the failures on other PRs. Though if I have to choose, I think seeing the failure and manually waiving would be more important than skipping something altogether. Let me know if I misread your question.
Now that you mention it, more granular level is probably also desirable. In the PR I linked as example, all CentOS Stream jobs are failing so that's why I had the entire job in mind, but if it's only centos-stream-9 failing and centos-stream-10 passing, then waiving for each target would be preferable. |
By The first option is a bit more wasteful but probably easier to do. The second might have some issues on GitLab. Regarding GitLab, we tried to be forge-independent as much as possible but sometimes, it's just not possible -- e.g. check runs are GitHub specific. |
Thanks! My idea was that if we do a new run as a reaction to the comment, we could just not run the job that is waived but it won't work because the waived result will be preserved on the GitHub UI. (We need to reset this status to get rid of the failure.) |
If we were able to specify the tests to waive statically, we could define this just on the tmt level. Alternatively, we can think about a generic way how to (re)define tmt context when running a comment command but it would be hard to provide a nice syntax. Especially if you would need this for a more granular level. |
How complicated would it be to |
@lsm5 Do you mean to re-run the test suite on GitLab and change the status to neutral on GitHub? I think it's doable but maybe I would slightly prefer to behave in the same way. |
Yes
SGTM. Thanks @lachmanfrantisek |
Description
In podman and related tools, we're running build and test jobs on both RHEL and CentOS Stream. Often either RHEL or CentOS Stream tests may fail depending on underlying issues like
Many of these failures are non-blocking and often unrelated to the PR submitted.
What I'm looking for:
/packit waive foo-test
, where foo-test could be a value for either atest identifier
or thepackages
key.Example: containers/buildah#5514 where centos stream tests are failing because of changes in golang that haven't yet landed in RHEL. Over there, I'd love to be able to comment
/packit waive buildah-centos
and then have the red flag on those tests changed to a neutral color maybe with aWaived
status against it.Benefit
This feature request follows the Fedora bodhi model of running all tests and then waiving known non-blocking issues.
I suspect depending on the
label
approach with manual triggers might lead to tests not being run because people forgot or were simply not aware.Importance
It's important in the sense that it will:
What is the impacted category (job)?
General
Workaround
Participation
The text was updated successfully, but these errors were encountered: