[v5.4-rhel] Enable system tests on RHEL envs with TMT

[lsm5]

Does this PR introduce a user-facing change?

None

[packit-as-a-service]

Failed to load packit config file:

Cannot parse package config. ValidationError({'jobs[0].packages': 'Undefined package(s) referenced: podman-rhel.', 'jobs[1].packages': 'Undefined package(s) referenced: podman-rhel.'})

For more info, please check out the documentation or contact the Packit team. You can also use our CLI command config validate or our pre-commit hooks for validation of the configuration.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lsm5

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [lsm5]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[packit-as-a-service]

Ephemeral COPR build failed. @containers/packit-build please check.

[packit-as-a-service]

Tests failed. @containers/packit-build please check.

[packit-as-a-service]

Ephemeral COPR build failed. @containers/packit-build please check.

[Luap99]

We either must reconfigure merge protection to be turned off or hard code the tasks names there to the tmt ones for all rhel branches. I guess for RHEL branches the task names don't change so setting up branch protection and for each branch will be acceptable. (And I guess long term we wanted to drop the RHEL branches from the main repo here anyway so this would not be a forever situation)

Also why are we testing epel and nightly branches, would it not make much more sense to target the actual branches this version is being shipped in? I would expect to be tested against RHEL 9.6/10.0 only. That way we know the right go version is being used and the tests pass on the final version that should massively reduce any post merge surprises.

[Luap99]

Oh on other thing there is must include Jira link for RHEL CI setup via cirrus via the auto team. I am not sure what our commitment there is/was but I think we must have a way to ensure that only RHEL Jira approved changes can get merged.

[lsm5]

We either must reconfigure merge protection to be turned off or hard code the tasks names there to the tmt ones for all rhel branches. I guess for RHEL branches the task names don't change so setting up branch protection and for each branch will be acceptable. (And I guess long term we wanted to drop the RHEL branches from the main repo here anyway so this would not be a forever situation)

Agreed.

Also why are we testing epel and nightly branches, would it not make much more sense to target the actual branches
this version is being shipped in?

RE: epel, the way copr is setup, epel jobs are the most convenient for RHEL testing. The copr epel targets are RHEL + EPEL.

RE: nightly compose, the current rhel-10-main and rhel-9-main have v5.4, so I've enabled them here atm.

I would expect to be tested against RHEL 9.6/10.0 only. That way we know the right go version is being used and the tests pass on the final version that should massively reduce any post merge surprises.

Yup, I'll enable the other relevant targets too.

Eventually, it would be good to add these jobs in the main branch along with Packit's PR-label-based job triggers, which would make it a lot easier to enable these jobs whenever we cut a new rhel maint branch, instead of adding such a change separately to every new rhel branch.

Oh on other thing there is must include Jira link for RHEL CI setup via cirrus via the auto team. I am not sure what our commitment there is/was but I think we must have a way to ensure that only RHEL Jira approved changes can get merged.

I'll look into that. Thanks.

[lsm5]

Eventually, it would be good to add these jobs in the main branch along with Packit's PR-label-based job triggers, which would make it a lot easier to enable these jobs whenever we cut a new rhel maint branch, instead of adding such a change separately to every new rhel branch.

I'm also checking with the packit team about additional workflow options which could possibly help us maintain the official rpm through the rhel maint branch itself instead of having 2 separate repos. If we move hosting for rhel branches elsewhere, then of course my quoted comment won't be relevant.

[Luap99]

Eventually, it would be good to add these jobs in the main branch along with Packit's PR-label-based job triggers, which would make it a lot easier to enable these jobs whenever we cut a new rhel maint branch, instead of adding such a change separately to every new rhel branch.

I mean we talk about a branch once every six months so I would not worry about that.

If we move hosting for rhel branches elsewhere, then of course my quoted comment won't be relevant.

I think we must, given CNCF we cannot really have RHEL specific code in the same repo. Basic access rule would not work out. And I think that is actually a strong point in favour of tmt. Because we can easily switch to the centos gitlab or whatever and keep using the same packit/tmt config and infra I assume. And then we can have proper merge protection on such new repo for tmt based tasks.

[packit-as-a-service]

Failed to load packit config file:

Please correct data and retry.

For more info, please check out the documentation or contact the Packit team. You can also use our CLI command config validate or our pre-commit hooks for validation of the configuration.

[lsm5]

@containers/podman-maintainers PTAL

[lsm5]

No rpms are being built here. Just makefile targets as-is.

[Luap99]

Why doesn't this use the actual rpm build?
I think that is likely causing some of your testing issues. I would assume if this is using podman-tests like done on main/gating tests.

[lsm5]

That's because using centos stream spec causes problems, mainly centos stream moving to v5.5 etc. and rhel specs aren't accessible. Which is why my comment on the tracker issue about finding a new home and change in process for RHEL branches.

I would have ideally preferred to put this whole thing on hold until we do the process change, but this at least gets us close to cirrus independence.

Packit is not available internally, so if we are moving the rhel branches to somewhere internal, the process change would need us to write our own build rpm job, or borrow the one that's used on centos stream dist-git. So, having some rpm job here would not easily transfer to the new home. So, if we don't use rpms here, it becomes much easier to move rhel branches elsewhere, as TMT jobs get triggered with or without packit.

[Luap99]

ack go it, yeah it seems we are a in bit of weird spot right now.

I guess problems will get solved once we have a spec in repo + packit/tmt workflow for RHEL properly worked out, most likely means having an internal repo.

I am ok moving ahead with this but I suggest you ask QE to check all the newly skipped issues so we know if they actually test them or know how to make them pass in a RHEL env.

[lsm5]

I guess problems will get solved once we have a spec in repo + packit/tmt workflow for RHEL properly worked out, most likely means having an internal repo.

The in-repo spec copy was one option I had in mind, but that could suffer from rot because we don't use the packit process for RHEL and there's risk of that spec being ignored/diverging from official spec. So, ideally it should be a spec that's regularly maintained in the official RHEL process and also easily fetchable.

It would be ideal to have the official rhel sources and the podman RHEL branch be a monorepo, so everyone could work on the same repo, but that process isn't in place yet.

I am ok moving ahead with this but I suggest you ask QE to check all the newly skipped issues so we know if they actually test them or know how to make them pass in a RHEL env.

On it. I'm gonna try running these on their environment tomorrow. Yiqiao said the rootless user is added to wheel in their env, but when I add rootless user to wheel as in 3b6d20c . it skips all of 252-quadlet.bats as in https://artifacts.osci.redhat.com/testing-farm/9b8695bd-3651-45f5-80d0-7505b7b90df7/ . I'll get back to this tomorrow.