e2e-upgrades: add basic tests with new rpm-ostree version

[HuijingHei]

Test new rpm-ostree with kernel override, initramfs args, initramfs-etc, layering, overrides.
See #4776

[HuijingHei]

/retest

[openshift-ci]

@HuijingHei: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/fcos-e2e	de79076	link	true	/test fcos-e2e
ci/prow/kola-upgrade	de79076	link	true	/test kola-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[HuijingHei]

Confused about the error error waiting for pod: Get "https://api-int.build02.gcp.ci.openshift.org:6443/api/v1/namespaces/ci-op-m1kkigtc/pods/kola-upgrade-kola-upgrades?timeout=1m0s": tls: failed to verify certificate: x509: certificate signed by unknown authority, does this mean can not reach https://koji.fedoraproject.org/koji/buildinfo?buildID=2302642 ?

[jlebon]

Thanks for working on this!

Three things from my side:

1. IIUC, the first boot is already working from the new rpm-ostree (via --oscontainer passed to kola), but I think what we want is to create state from a node on the old rpm-ostree and then update on the new rpm-ostree.
2. Confusingly, IIUC the second boot is actually working from the old rpm-ostree, which is an interesting test (it verifies that state created by a new rpm-ostree does not break an old one). But a more natural test is upgrading to another update with the new rpm-ostree so that you can verify that new features are working correctly.
3. Totally OK if you want to pursue this via the Prow harness. I personally think making this part of the other kola tests covered by CoreOS CI would make it more accessible and easier to maintain.

To fix 1, you'd probably want in the first boot to rollback to e.g. what's in testing-devel right now before creating state. To fix 2, you can do it the container way (export current commit to a container image, then create a new container image with some arbitrary change in it -- see e.g. this test) or the OSTree way (create a synthetic commit and rebase onto it -- see e.g. this test)

[HuijingHei]

1. IIUC, the first boot is already working from the new rpm-ostree (via --oscontainer passed to kola), but I think what we want is to create state from a node on the old rpm-ostree and then update on the new rpm-ostree.

Yes, this is to add some basic tests with new rpm-ostree (via --oscontainer) in the old build fcos-stable, then update using new rpm-ostree. But seems there is problem with https, should consider about your option 3 about kolainst.

2. Confusingly, IIUC the second boot is actually working from the old rpm-ostree, which is an interesting test (it verifies that state created by a new rpm-ostree does not break an old one). But a more natural test is upgrading to another update with the new rpm-ostree so that you can verify that new features are working correctly.
3. Totally OK if you want to pursue this via the Prow harness. I personally think making this part of the other kola tests covered by CoreOS CI would make it more accessible and easier to maintain.

To fix 1, you'd probably want in the first boot to rollback to e.g. what's in testing-devel right now before creating state. To fix 2, you can do it the container way (export current commit to a container image, then create a new container image with some arbitrary change in it -- see e.g. this test) or the OSTree way (create a synthetic commit and rebase onto it -- see e.g. this test)

Will look at it, thanks a lot @jlebon for your quick reply!

Maybe we can have 2 tests about the upgrades, does this make sense?

• According to your suggestion: Do some basic tests (like kernel override, initramfs args, initramfs-etc, layering, overrides) in the old builds like fcos-stable, then upgrade new rpm-ostree, then upgrade to another update.
• Upgrade rpm-ostree in old builds like fcos-stable, then do some basic tests (see above) using the new rpm-ostree, then upgrade to another update.

[jlebon]

Maybe we can have 2 tests about the upgrades, does this make sense?

* According to your [suggestion](https://github.com/coreos/rpm-ostree/issues/4776): Do some basic tests (like kernel override, initramfs args, initramfs-etc, layering, overrides) in the old builds like fcos-stable, then upgrade new rpm-ostree, then upgrade to another update.

* Upgrade rpm-ostree in old builds like fcos-stable, then do some basic tests (see above) using the new rpm-ostree, then upgrade to another update.

I.e. in one test we do the mutations on top of old rpm-ostree and then upgrade, and in the other test, we upgrade first and then do the mutations? The first one sounds more interesting to me. What do you have in mind on what the second one is intended to catch?

[HuijingHei]

Maybe we can have 2 tests about the upgrades, does this make sense?

* According to your [suggestion](https://github.com/coreos/rpm-ostree/issues/4776): Do some basic tests (like kernel override, initramfs args, initramfs-etc, layering, overrides) in the old builds like fcos-stable, then upgrade new rpm-ostree, then upgrade to another update.

* Upgrade rpm-ostree in old builds like fcos-stable, then do some basic tests (see above) using the new rpm-ostree, then upgrade to another update.

I.e. in one test we do the mutations on top of old rpm-ostree and then upgrade, and in the other test, we upgrade first and then do the mutations? The first one sounds more interesting to me. What do you have in mind on what the second one is intended to catch?

The second is to avoid regression of fedora-silverblue/issue-tracker#523, which upgrade rpm-ostree on old build, but hit issue when remove package.

[HuijingHei]

Close this as impl this in #4806