tests: add rpm-ostree override remove test

[HuijingHei]

Add integration test for fedora-silverblue/issue-tracker#523

[HuijingHei]

The regression issue can be reproduced on rawhide 40.20231220.91.0, which is built using coreos-assembler with rpm-ostree version: 2023.11, but inside the OS image contains rpm-ostree-2023.12-1.fc40.x86_64 (fedora-rawhide).

See https://jenkins-fedora-coreos-pipeline.apps.ocp.fedoraproject.org/job/build/2137/consoleText

Add test to avoid future regression.

[cgwalters]

The version of rpm-ostree used by cores-assembler isn't important for catching this bug. What we need to have skewed is the version using in the target OS. And it needs to work like this I believe to reproduce the failure:

• start from the old version (e.g. boot fcos stable)
• do the override remove
• upgrade to a version with the new rpm-ostree

I think in the general case we may also need to try upgrading again after that.

[HuijingHei]

The version of rpm-ostree used by cores-assembler isn't important for catching this bug. What we need to have skewed is the version using in the target OS. And it needs to work like this I believe to reproduce the failure:

• start from the old version (e.g. boot fcos stable)
• do the override remove
• upgrade to a version with the new rpm-ostree

I think in the general case we may also need to try upgrading again after that.

Thanks! Sounds more reasonable, does it makes sense to add the test in rpm-ostree git repo instead of here?

[jlebon]

Yeah, this probably belongs better in the rpm-ostree repo. In general, we don't have a lot of rpm-ostree-specific upgrade tests like this. But it certainly would make sense to do so.

We could even have a single test that leverages every client-side layering feature (e.g. kernel override, initramfs args, initramfs-etc, layering, overrides, etc...) and verify that it can upgrade to the test deployment (composed with and containing the PR code), and then upgrade again to some new target.

[HuijingHei]

Thanks @cgwalters @jlebon for the comment, will close this and try with rpm-ostree repo.

[cgwalters]

It'd still make sense in the future to support running tests that live in rpm-ostree.git as part of other testing runs (i.e. not just on PRs to that repo), which is part of the general thread in the Fedora+derivatives space of integration with https://docs.fedoraproject.org/en-US/ci/manifesto/ and running in dist-git merges etc.